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GLOBAL  TEAMS:  ENHANCING  THE  PERFORMANCE  OF  MULTINATIONAL  STAFFS 
THROUGH  COLLABORATIVE  ONLINE  TRAINING 

EXECUTIVE  SUMMARY  


Research  Requirement: 

Developing  and  delivering  high-quality  training  for  the  variety  of  contingencies  that  may 
arise  in  the  contemporary  operating  environment  is  vital  to  preparing  a  stronger  Future  Force. 
Currently  there  is  a  need  for  more  research  to  identify  the  key  challenges  in  the  operational 
environment  and  for  the  development  of  focused  and  relevant  training  that  can  support  improved 
decision  making  and  coordination.  To  prepare  for  the  challenges  of  coalition  operations,  cost- 
effective  training  is  needed  to  build  trust  and  improve  decision  making  and  coordination  among 
coalition  partners.  There  were  two  primary  objectives  for  this  research  effort.  The  first  was  to 
develop  high  quality  training  to  build  expertise  in  coalition  coordination  and  decision  making  by 
supporting  participants’  awareness  of  the  key  challenges  of  coalition  coordination  and  how  they 
can  be  overcome  in  mission  specific  circumstances.  The  second  major  objective  was  to  explore 
the  feasibility  of  online  collaboration  platforms  as  a  training  medium  for  coalition  operations. 

Procedure: 

We  used  the  considerable  operational  experience  of  English-speaking  (American,  British, 
Canadian,  Australian)  coalition  officers  to  identify  the  key  cognitive  challenges  inherent  in 
coalition  operations.  The  key  challenges  were  used  to  drive  the  development  of  context-rich 
scenarios  and  to  guide  development  of  a  training  program.  To  facilitate  the  training,  we 
developed  a  collaborative  web-based  distributed  training  tool  that  would  allow  at  least  two 
coalition  partners,  in  addition  to  a  facilitator,  to  practice  working  together  as  a  coalition  staff 
using  a  scenario-based  training  program. 

Findings: 

The  evaluation  objective  was  to  explore  the  feasibility  of  online  collaboration  platforms 
as  a  training  medium  for  coalition  operations.  As  a  result  of  the  evaluation,  six  critical  factors 
were  identified,  which  impact  the  effectiveness  of  the  training.  1.  Clarify  the  learning  objectives 
in  advance;  2.  Emphasize  the  problem  solving  and  coordination  aspects  of  the  exercise;  3. 
Capitalize  on  the  opportunity  for  interaction  by  allowing  partners  to  interact  over  discussion  and 
problem  solving;  4.  Set  the  training  at  the  appropriate  level;  5.  Use  an  experienced  facilitator  to 
direct  the  training;  6.  Tie  tool  functionality  to  the  learning  strategy. 

Utilization  and  Dissemination  of  Findings: 

Overall,  the  training  was  shown  to  improve  participants’  awareness  of,  and  ability  to  respond  to, 
the  key  themes  of  coalition  coordination.  Briefings  were  provided  to  the  Joint  Readiness 
Training  Center  (JRTC)  Operations  Group,  September  7,  2005,  and  to  the  Joint  Services 
Command  and  Staff  College  (JSCSC),  March  1, 2005.  The  developed  method  and  tool  provide 


an  easy-to-implement  and  efficient  means  for  coalition  partners  to  train  in  a  distributed 
environment,  on  short  notice,  prior  to  deploying  to  the  operating  environment. 
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THROUGH  COLLABORATIVE  ONLINE  TRAINING 


Introduction 

Coalition  operations  have  emerged  as  a  key  feature  of  military  operations  in  the  post- 
Cold  War  era.  While  the  U,S.  Army  has  participated  in  coalition  operations  throughout  its 
existence,  it  is  now  the  standard  mode  across  the  spectrum  of  military  interventions.  “Virtually 
all  U.S.  military  operations  since  1989  have  been  conducted  as  part  of  a  multinational  operation” 
(Bensahel,  2003,  p.  112).  A  coalition  operation  usually  entails  an  ad  hoc  arrangement  between 
two  or  more  nations  acting  together  in  order  to  pursue  a  common  objective  American-British- 
Canadian-Australian  ([ABCA],  2001).  They  are  formed  where  a  commonality  of  interests  exists 
between  nations,  be  it  political,  economic,  or  military,  allowing  those  nations  the  benefit  of 
mutual  aid  in  promoting  their  national  interest  and  securing  against  real  or  perceived  threats. 
They  are  usually  created  for  specific  purposes  and  for  a  limited  duration.  To  date,  the  English 
speaking  coalition  nations,  the  focus  of  this  report,  have  served  together  in  ad  hoc  coalitions  on 
several  occasions  to  pursue  common  objectives  (ABCA,  2001). 

Coalition  operations  are  very  different  from  unilateral  operations.  They  engender  a 
number  of  organizational,  operational,  political,  and  cultural  challenges  which  are  unique  to  the 
operational  environment.  Organizationally,  coalitions  can  be  very  complex.  They  often  include 
various  branches  of  the  U.S.  military  and  foreign  militaries,  as  well  as  nonmilitary  organizations 
such  as  non-governmental  organizations  (NGOs)  and  private  voluntary  organizations  (PVOs), 
each  bringing  their  own  unique  core  competencies  and  political  legitimacy.  This  blending  of 
capabilities  makes  possible  certain  operations  that  a  single  nation  could  not,  or  would  not 
conduct  unilaterally.  However,  coalitions  also  engender  significant  coordination  challenges.  For 
example,  consensus  can  be  difficult  to  achieve  because  national  interest  may  compete  with 
standard  practice  as  outlined  in  doctrine  (Joint  Chiefs  of  Staff,  2000). 

Operationally,  coalitions  cross  the  entire  range  of  military  operations  from  combat  to 
humanitarian  to  stability  and  support  operations  (SASO).  The  precise  role  of  armies  in  these 
operations  varies  according  to  each  political  and  military  situation.  In  some  situations  military 
operations  are  subordinate  to  political  and  diplomatic  efforts.  In  such  circumstances,  coalitions 
have  to  be  very  creative  to  find  ways  to  deliver  appropriate  and  effective  levels  of  military 
leverage  that  support  political-diplomatic  initiatives  within  the  prescribed  limitations  (Peters, 
Johnson,  Bensahel,  Liston  &  Williams,  2001).  In  other  situations,  some  coalition  partners  may 
participate  in  the  humanitarian  and  peacekeeping  phases  of  an  operation,  but  not  in  combat, 
should  that  occur.  Moreover,  because  of  their  limited  purpose  and  life  span,  coalitions  do  not 
afford  the  same  political  resolve  and  commonality  of  aim  as  other  types  of  military  operations. 
Within  this  realm,  sovereignty  issues  are  often  the  most  difficult  issues  for  coalition  forces  to 
deal  with.  Often,  the  commander  of  a  coalition  force  is  a  commander  in  title  only;  mission 
accomplishment  is  achieved  through  coordination,  communication,  and  consensus  rather  than  by 
traditional  command  concepts  (i.e.,  giving  an  order)  (Joint  Chiefs  of  Staff,  2000). 

Political  and  sovereignty  issues  make  planning  and  coordination  of  coalition  operations 
particularly  difficult.  Commanders  and  planners  must  learn  the  capabilities  of  partner  nations  or 
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organizations  and  must  design  plans  in  such  a  way  as  to  allow  all  units  to  contribute  (ABCA, 
2001).  In  many  cases,  the  coalition  leadership  will  have  to  be  innovative  to  find  effective  roles 
for  all  participants  (Peters  et  al.,  2001 ).  Army  doctrine  emphasizes  the  importance  of  including 
NGO  and  PVO  representatives  at  the  earliest  possible  phase  of  mission  planning.  The  sensitivity 
and  understanding  gained  by  including  these  organizations  early  in  the  planning  process  can 
contribute  to  more  effective  working  relationships  once  the  mission  begins  (Joint  Chiefs  of  Staff. 
2000). 


Coalition  doctrine  also  recognizes  the  role  of  intangible  factors  in  multinational 
collaboration.  In  particular,  developing  effective  partnerships,  confidence,  rapport  and  respect 
are  considered  to  be  the  cornerstones  of  successful  coalition  operations.  In  fact,  after  World  War 
II,  General  Dwight  D.  Eisenhower  said  that  "mutual  confidence”  is  the  "one  basic  thing  that  will 
make  allied  commands  work”  (Joint  Chiefs  of  Staff,  2000,  p.  1-9). 

Good  rapport  between  coalition  members  results  in  successful  teamwork  and  overall 
unity  of  effort  (Joint  Chiefs  of  Staff,  2000).  The  first  concern  when  establishing  rapport  is  an 
understanding  of  the  characteristics,  personalities,  capabilities,  ambitions,  and  cultural  habits  of 
the  various  coalition  partners.  Rapport  is  more  easily  established  when  nations  combining  forces 
share  similar  cultural  backgrounds.  Respect  for  the  partners'  culture,  religion,  customs,  and 
values  combined  with  understanding  and  consideration  of  their  ideas  will  solidify  the 
partnership.  Without  such  genuine  respect  of  others,  rapport  and  mutual  confidence  cannot  exist; 
lack  of  respect  may  lead  to  friction,  jeopardizing  mission  accomplishment.  While  these  factors 
cannot  guarantee  success  for  the  coalition,  ignoring  them  can  usually  guarantee  failure  of  the 
coalition  in  accomplishing  its  mission  (Joint  Chiefs  of  Staff,  2000). 

Commanders  must  possess  the  knowledge  and  experience  that  is  critical  for  effective 
decision  making  and  teamwork  in  these  types  of  operations.  The  key  challenge  for  commanders 
appears  to  be  coordinating  with  coalition  partners,  international  and  civilian  organizations,  and 
the  host  nation  government  and  local  citizens.  The  literature  describes  this  as  a  problem  of 
"interoperability.”  To  date,  a  clear  understanding  of  how  to  improve  interoperability  in  these 
operations  has  not  yet  been  established  (ABCA,  2001;  Ryan,  2000).  For  example,  what  is  the 
nature  of  effective  coordination  among  such  diverse  players?  What  are  the  organizational 
arrangements  by  which  coalition  operations  solve  problems?  Who  needs  to  interface  with 
whom,  by  what  means  do  they  do  it,  and  what  are  the  critical  timeframes  for  particular  kinds  of 
communications  among  particular  players?  How  should  commanders  approach  decision  making 
in  peace  operations  with  limited  resources  and  restrictions  on  unity  of  command?  What 
opportunities  must  commanders  create  for  themselves  in  the  face  of  seemingly  impossible 
conditions?  These  and  other  critical  questions  about  the  judgments,  decisions,  creativity, 
organizational  arrangements,  and  control  faced  by  coalition  commanders  need  to  be  identified 
and  answered  from  the  perspective  of  the  commander. 

Multinational  exercises  are  thought  to  be  extremely  important  for  ensuring 
interoperability  with  potential  coalition  partners  and  for  working  out  command  issues. 

Moreover,  researchers  believe  that  NATO  should  improve  its  capability  for  exercises  so  that  it 
can  routinely  integrate  coalition-building  and  maintenance  activities  with  military  action  (Peters 
et  at,  2001). 
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Identification  of  Training  Need 


Developing  and  delivering  high-quality  training  for  the  variety  of  contingencies  that  may 
arise  in  the  contemporary  operating  environment  is  vital  to  preparing  a  strong  Future  Force. 
Indeed  the  U.S.  Army  planning  documents  for  the  Future  Combat  System  of  Systems 
specifically  call  out  the  requirement  to  “develop,  through  training  and  experience,  the  thinking, 
confident,  versatile,  adaptive,  and  seasoned  leaders  at  the  tactical  level  required  for  the  digitized, 
rapidly  deployable  objective  force”  (U.S.  Army  Training  and  Doctrine  Command,  2001 ,  p.  6- 
29).  In  order  to  achieve  this  objective  there  is  a  recognized  need  for  more  research  to  identify 
the  key  challenges  in  this  operational  environment,  and  for  the  development  of  focused  and 
relevant  training  that  can  support  improved  decision  making  and  coordination. 

Research  Objectives 

There  were  two  primary  objectives  for  this  research  effort.  The  first  was  to  develop  high 
quality  training  to  build  expertise  in  coalition  coordination  and  decision  making  by  supporting 
participants'  awareness  of  the  key  challenges  of  coalition  coordination  and  how  they  can  be 
overcome  in  mission  specific  circumstances.  The  second  major  objective  was  to  explore  the 
feasibility  of  online  collaboration  platforms  as  a  training  medium  for  coalition  operations.  The 
training  will  involve  the  development  of  low-fidelity  scenario-based  exercises.  Multiple, 
distributed  participants  will  be  required  to  make  a  decision  or  complete  a  task  based  on  a 
cognitively  authentic  scenario-based  challenge. 

Developing  Training  for  Multinational  Coalition  Operations  Teams 

This  effort  involved  developing  training  requirements  for  multinational  command  and 
staff  teams  at  the  brigade  and  multinational  division  level,  designing  and  implementing  scenario- 
based  training,  and  evaluating  the  distributed  multinational  training  for  feasibility  and 
effectiveness.  Scenarios  are  based  on  real-world  situations  to  provide  participants  access  to  the 
expertise  required  to  address  such  challenges.  Our  goal  was  not  to  develop  cultural  training,  per 
se,  but  to  address  the  differences  in  how  English-speaking  nations1  set  goals  and  work  to  achieve 
them,  as  well  as  to  highlight  the  differential  strengths  they  bring  to  coalition  operations. 

Scenario-based  training  has  many  advantages  over  factual,  classroom  training.  It  allows 
the  students  to  assess  situations,  identify  relevant  facts,  strategies,  and  procedures,  generate 
possible  solutions,  make  decisions,  and  receive  natural  and  instructional  feedback.  Whereas 
classroom-based  training  is  about  gathering  information,  scenario-based  training  is  about 
doing — making  decisions,  planning,  and  applying  the  knowledge  and  information  for  making 
decisions  in  an  operational  scenario.  This  training  provides  an  opportunity  to  develop  a  more 
detailed  mental  model  of  how  aspects  of  a  particular  domain  actually  work  by  working  through 
the  issues  and  interplay  of  factors  through  realistic  problem  scenarios.  It  is  recognized  that  the 
Army  uses  tactical  decision  exercises  and  scenario-based  training  in  the  classroom  on  a 
consistent  basis,  however,  the  training  approach  presented  here  has  the  added  benefit  of  allowing 
real  time  problem  analysis,  coordination  and  problem  solving  with  multinational  partners. 


1  This  research  was  supported  by  the  Technical  Cooperation  Program  (http://www.dtic.mil/ttcpO- 
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Our  research  methodology  is  based  on  a  combination  of  two  training  programs,  both  of 
which  are  grounded  in  the  principles  of  Recognition  Primed  Decision  (RPD)  making  and 
expertise  development  (Klein,  1989).  The  Think  Like  A  Commandeer  (TLAC)  training  program 
is  based  on  an  understanding  of  how  expertise  is  developed  and  embodies  the  concept  of 
deliberate  practice  of  thinking  skills  (Ross  &  Lussier,  1999).  It  aims  to  develop  the  learner  along 
a  path  towards  expertise  by  both  helping  him  elaborate  his  mental  models  and  by  making  the 
results  of  the  recognitional  decision  making  routines  explicit  so  that  they  can  be  refined.  A 
central  component  of  the  TLAC  program  is  the  use  of  context-rich  training  vignettes.  The  goal 
of  TLAC  is  to  teach  officers  “how  to  think"  instead  of  “what  to  think”  (Shadrick  &  Lussier, 
2002). 


The  other  training  program  informing  our  training  methodology  is  the  Decision  Skills 
Training  Program.  This  training  program  is  based  on  an  understanding  of  how  expertise 
develops  and  functions  in  dynamic  settings.  The  first  task  is  to  “unpack”  the  knowledge  base  of 
experts  in  a  field  so  that  the  knowledge  is  available  to  training  developers.  Cognitive  task 
analysis  methods,  primarily  the  Critical  Decision  Method  (Crandall  &  Getchell-Reiter,  1993; 
Hoffman,  Crandall,  &  Shadbolt,  1998),  are  used  to  leverage  the  critical  events  (stories)  from 
experts,  and  by  so  doing  expose  their  domain  expertise.  Second,  stories  (vignettes)  are  carefully 
constructed  to  include  realistic  cues  in  the  context.  The  incidents  collected  from  experts  arc  not 
used  intact.  Instead,  elements  are  emphasized  and  tailored  to  sharpen  the  focus  of  desired 
features.  All  of  the  vignettes  present  a  dilemma  with  some  level  of  uncertainty  and  a 
requirement  that  forces  the  student  to  make  a  decision.  Each  participant  has  a  limited  amount  of 
time  to  consider  how  he  or  she  would  react,  which  adds  time  pressure  to  the  exercise. 

Facilitation  ensures  that  the  participants  do  not  just  have  an  experience,  but  that  they  focus  on  the 
cognitive  aspects  of  that  experience — that  they  consider  the  depth  of  their  understanding  and  the 
likely  consequences  of  their  decisions. 

The  training  programs  described  above  are  somewhat  different  in  execution,  but  they 
both  support  the  development  of  mental  models,  the  acquisition  of  knowledge  in  context,  and  the 
practice  of  cognitive  activities  with  coaching.  Both  take  the  learner  further  along  the  path  to 
“expert  intuition.”  Moreover,  at  the  heart  of  both  of  these  training  methodologies  is  a  carefully 
crafted  story.  What  is  to  be  learned  and  practiced  has  been  made  explicit  and  incorporated  by  the 
training  developers.  Stories  represent  a  powerful  means  of  transferring  expertise  and  allowing 
the  learner  to  activate  that  expertise  in  new  situations.  The  central  challenge  is  that  of  creating 
training  scenarios  that  represent  authentic  and  believable  situations  and  represent  accurately  the 
task  work,  teamwork,  and  communication  tasks  that  are  needed  to  enable  the  trainee  to  practice 
the  needed  tasks  and  skills. 

Requirements  Analysis  and  Training  Development 

This  effort  was  completed  in  four  phases:  Phase  1 :  Data  Collection  to  Identify  Training 
Requirements;  Phase  2:  Training  Requirements  Analysis  and  Scenario  Development;  Phase  3: 
Developing  the  Learning  Strategy  and  Training  Method;  and  Phase  4:  Training  Evaluation. 
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Phase  1:  Data  Collection  to  Identify  Training  Requirements 

The  data  collection  phase  had  two  aspects  to  it:  domain  analysis  and  Critical  Decision 
Method  (CDM)  interviews.  Each  is  briefly  described  below. 

Domain  Analysis 

The  purpose  of  the  domain  analysis  process  was  to  support  a  better  understanding  of  the 
mission  and  tasks  that  are  involved  in  multinational  operations,  particularly  those  that  are 
cognitively  challenging  or  in  need  of  training.  A  secondary  aim  was  to  solicit  the  subject  matter 
experts'  (SME)  suggestions  about  fruitful  avenues  for  scenario  development.  The  SMEs  were  a 
British  officer  and  a  Canadian  officer  stationed  at  Fort  Knox.  Both  were  Foreign  Liaison 
Officers  and  had  extensive  experience  with  coalition  operations.  The  third  SME  was  a  U.S. 
Lieutenant  Colonel  (LTC),  (Active  Duty  Reservist)  with  25  years  of  military  coalition 
experience. 

Critical  Decision  Method  Interviews 

Critical  Decision  Method  (CDM)  interviews  (Hoffman,  Crandall.  &  Shadbolt,  1998; 
Militello  &  Hutton,  1998;  Schraagen,  Chi  pm  an.  &  Shalin,  2000)  were  conducted  to  identify  the 
decision  requirements  in  coalition  operations  in  the  contemporary  operating  environment. 

Critical  Decision  Method  interviews  were  conducted  with  a  total  of  16  SMEs.  Of  those,  seven 
interviews  were  conducted  with  SMEs  who  were  Majors  at  the  Command  and  General  Staff 
College  (CGSC),  Fort  Leavenworth,  (six  were  U.S.  Army  officers,  one  was  a  Canadian  Army 
officer).  The  remaining  nine  CDM  interviews  were  conducted  at  Joint  Services  Command  Staff 
College  (JSCSC)  in  England  with  UK  Army,  Air  Force,  and  Navy  officers.  Officers  ranks  (or 
equivalents)  were:  Brigadier  General  (BG)  (2),  Colonel  (COL)  (2),  Lieutenant  Colonel  (LTC) 
(2),  and  Major  (MAJ)  (3). 

The  CDM  interviews  were- organized  around  an  initial,  unstructured  account  of  a  specific 
incident  experienced  by  the  interviewee.  In  this  case,  participants  were  asked  to  recall  an 
incident  that  involved  collaboration  with  a  coalition  partner  that  they  found  to  be  particularly 
challenging.  Once  the  participant  identified  a  relevant  incident,  he  or  she  recounted  the  episode 
in  its  entirety,  with  no  interruptions  from  the  interviewer.  The  initial  incident  account  provided 
the  focus  and  structure  for  the  remainder  of  the  interview.  Once  the  report  of  the  incident  was 
completed,  the  CDM  interviewer  led  the  participant  back  over  his  or  her  incident  account  several 
times,  using  probes  designed  to  focus  attention  on  particular  aspects  of  the  incident  and  solicit 
information  about  them.  The  probes  were  designed  to  progressively  deepen  understanding  of  the 
interviewee's  account.  The  information  obtained  via  these  methods  is  concrete  and  specific, 
reflects  the  point  of  view  of  the  decision  maker,  and  is  grounded  in  actual  incidents.  This 
method  has  been  demonstrated  to  yield  information  richer  in  variety,  specificity,  and  quantity 
than  is  typically  available  in  experts’  unstructured  verbal  reports.  Detailed  descriptions  of  CDM 
and  the  work  surrounding  it  can  be  found  in  Klein  and  Hoffman  (1993)  and  Klein,  Calderwood, 
and  MacGregor  (1989). 
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Phase  2:  Training  Requirements  Analysis  and  Scenario  Development 

This  phase  invol  ved  analysis  of  the  interview  data  and  the  identification  of  High  Level 
Focus  Areas  {HLFAs)  and  their  associated  Awareness  Points  (APs).  A  description  of  how  these 
were  determined  is  outlined  in  the  next  section,  Training  scenarios  were  subsequently  developed 
which  addressed  the  HLFAs  identified  in  the  CDM  interviews. 

Training  Requirements  Analysis 

Interview's  were  coded  in  a  thematic  analysis  according  to  the  grounded  theory  approach 
(Glaser  &  Strauss,  1967).  A  process  of  classification  and  coding,  then  reclassification  and  re¬ 
coding,  led  to  the  development  of  a  classification  system  consisting  of  four  themes  or  HLFAs 
and  27  underlying  APs.  We  call  the  themes  HLFAs  because  they  guide  the  focus  of  the  training 
objectives  and  scenario  content.  The  HLFAs  are  the  areas  in  which  the  coalition  forces  differ  in 
operation.  Coalition  officers  must  develop  understanding  in  how  these  issues  can  impact 
operations.  Coalition  officers  without  such  an  understanding  may  have  less  successful 
collaborations.  This  is  evidenced  by  the  experiences  recounted  by  our  interview  participants. 

The  four  HLFAs  are:  organizational  structure,  work-style,  stance,  and  level  of  integration. 

Organizational  structure  refers  to  the  formal  and  informal  structures  that  govern  the 
organization  of  the  force.  This  includes  the  degree  of  autonomy  allowed  to  junior  officers  and 
their  ability  to  influence  decision  making  higher  up  the  organizational  hierarchy.  Awareness 
points  associated  with  this  HLFA  include  formal  chain  of  command,  force  structure,  informal 
chain  of  command,  and  mission  command.  The  HLFAs  and  APs  are  summarized  in  Table  1 . 

Work-style  refers  to  the  culture  and  climate  with  each  national  force.  It  includes  issues 
such  as  work  tempo,  task  focus,  and  the  preferred  medium  for  transmitting  information.  It  also 
includes  factors  which  relate  to  the  skill  sets  of  the  force  (i.e.,  how  adaptable  and  how  flexible 
they  are).  Awareness  points  associated  with  this  HLFA  include:  task  structure,  generalist  versus 
specialist,  task  versus  relationship,  tempo,  planning  process,  flexibility/adaptability,  and 
exchange  of  information. 

Stance  refers  to  attitudes  toward  force  protection  within  a  particular  force.  Related  to  this 
are  attitudes  toward  the  role  of  the  military  ‘in  country,”  the  overriding  objective  of  foreign 
missions,  attitudes  toward  host  nations  and  coalition  partners,  the  problem  solving  approach 
typically  adopted  by  a  nation,  and  how  fixed  or  flexible  it  is  in  using  alternative  methods  to  solve 
problems.  Awareness  points  associated  with  this  HLFA  include:  force  protection,  politics,  rules 
of  engagement  (ROE),  leverage  points,  problem  solving,  and  manage  presence. 

Level  of  integration  refers  to  the  level  of  awareness  between  coalition  partners;  how  well 
they  understand  each  other's  roles,  functions,  capabilities,  responsibilities,  approach  and  stance 
and  how  willing  they  are  to  collaborate  to  solve  problems.  Awareness  Points  associated  with  this 
HLFA  include:  understanding  your  coalition  partner(s),  understanding  the  host  nation’s  culture, 
understanding  “coalition,”  giving  direction,  information  exchange,  resources,  ROE,  priorities, 
trust,  understanding  NGOs,  roles  and  functions,  capabilities,  commander’s  intent.  Title  10,  and 
integration. 
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High  Level  Focus  Areas  and  Awareness  Points 


(Table  Continues) 


Scenario  Development 

The  project  team  developed  three  scenarios.  The  scenarios  were  based  on  the  lived 
experiences  of  coalition  forces  recounted  during  the  CDM  interviews.  Each  scenario  involves 
three  individual,  yet  connected  vignettes  that  build  on  each  other  to  form  evolving  coalition 
operational  challenges.  Vignettes  require  the  participants  to  interact  and  address  situations  from 
the  contemporary  operational  environment.  Each  vignette  addresses  one  or  more  of  the  HLFAs 
so  that  issues  that  impact  coalition  operations  can  be  worked  through  by  participants,  in  context, 
with  the  support  of  a  facilitator.  The  vignettes  are  designed  to  force  students  to  engage  in  a 
problem  solving  task.  The  student  will  be  required  to  explore  his/her  own  (or  a  partner's) 
assumptions,  approach,  goals,  mission,  objectives,  structure,  stance,  and  work -style,  and  to 
examine  the  impacts  of  these  on  task  and  mission  accomplishment. 

The  scenarios  are  summarized  below.  Full  versions  of  the  scenarios  are  available  in  the 
accompanying  compact  disc  (CD)  attachment  entitled  Global  Teams  Scenarios  and  Facilitation 
Guides.  They  are  also  available  online  at  http://www.361  interactive.com/gIobalteams/index.html. 

Scenario  1:  Quelling  the  Violence.  This  scenario  is  set  in  Iraq  around  the  time  of  the 
elections.  The  coalition  includes  United  Kingdom  (UK),  United  States  (U.S.)  Australia,  and 
New  Zealand  forces,  as  well  as  several  other  allied  nations.  The  theme  of  this  group  of  three 
exercises  is  about  dealing  with  requests  for  assistance  from  other  coalition  partners  and  how  the 
coalition  works  together  to  calm  an  insurgency.  The  players  take  the  role  of  staff  officers  at 
Combined  Joint  Task  Force  Headquarters  (CJTF  HQ)  in  Multi-National  Division  South  East 
(MND  SE).  As  the  country- wide  elections  approach,  there  is  great  political  pressure  on  all 
coalition  partners  to  ensure  that  the  insurgent  attacks  are  suppressed  so  that  Iraqi  citizens  can  go 
to  the  polls.  The  High  Level  Focus  Areas  (themes)  being  targeted  in  this  scenario  include: 
planning  tempo,  information  exchange,  integrating  resources  and  capabilities,  understanding  host 
nation  culture,  understanding  your  coalition  partner's  motivation,  objectives,  agenda  and  point  of 
view,  how  ROE  apply  to  the  mission,  political  motivation  for  participating  in  the  mission  and 
ramifications  of  actions,  identifying  things  in  the  environment  to  leverage  to  your  advantage, 
different  nations'  approach  to  solving  a  problem  and  understanding  your  coalition  partners' 
priorities.  In  Situation  1 ,  the  U.S.  has  asked  the  coalition  partners  to  support  ongoing  peace 
enforcement  operations.  Players  are  required  to  list  their  concerns  and  list  their  information 
requirements.  In  Situation  2,  coalition  forces  have  been  conducting  counter-insurgency  and 
peace  enforcement  operations.  Players  are  required  to  list  their  concerns,  what  they  plan  to  do, 
and  how  developments  may  change  the  way  coalition  operations  are  run  in  the  future.  Situation 
3  involves  a  hostage  taking  situation.  Players  are  required  to  list  their  concerns  and  state  how 
they  will  respond. 

Scenario  2:  Security  for  the  Refugee  Camp.  This  scenario  is  about  the  establishment  of  a 
refugee  camp  and  the  requirement  for  collaboration  between  forces  for  maintaining  camp 
security.  Coalition  partners  and  NGOs  are  working  together  to  set  up  refugee  camps  and  deal 
with  supporting  the  movement  of  large  numbers  of  displaced  persons.  The  players  are  members 
of  a  multinational  coalition  force  engaged  in  combat  and  counter-insurgency  operations.  The 
coalition  includes  the  UK,  the  U.S.,  Australia,  Poland,  and  New  Zealand,  as  well  as  several  other 
allied  nations.  The  High  Level  Focus  Areas  being  targeted  in  this  scenario  include:  understand 
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the  planning  process  of  your  coalition  partners,  identify  capabilities  and  limitations  of  your 
coalition  partners,  determine  how  to  integrate  assets  and  resources  to  accomplish  the  mission, 
determine  roles  and  functions,  and  identify  and  understand  potentially  different  perspectives  on 
security.  In  Situation  1 ,  the  UK.  forces  determine  that  they  do  not  have  sufficient  resources  to  set 
up  and  secure  a  refugee  camp  without  additional  reinforcement  from  their  coalition  partners. 
Players  are  required  to  decide  what  security  requirements  are  necessary  for  the  refugee  camp  by 
describing  their  three  top  priorities  for  securing  the  camp  and  describing  how  they  would  want  to 
allocate  U.S.  and  UK  security  assets.  In  Situation  2,  there  is  some  unrest  in  the  camp.  Players 
are  required  to  identify  possible  courses  of  action,  identify  considerations  to  take  into  account, 
and  identify-  what  roles  various  coalition  members  should  take.  In  Situation  3,  a  convoy  comes 
within  a  close  distance  to  the  group  of  hostile  Iraqis.  Players  are  required  to  decide  what 
advice/order  they  would  give  to  the  convoy  and  determine  their  top  three  concerns  about  the 
situation. 

Scenario  3:  Reconstruction  and  Compliance.  This  scenario  is  set  in  Afghanistan. 
Coalition  forces  include  the  U.S,,  the  UK,  Canada,  Australia,  New  Zealand,  and  Germany. 
Reconstruction  is  the  priority  of  this  coalition  effort.  Players  take  the  role  of  members  of  the 
Coalition  HQ  staff,  part  of  the  Coalition  Joint  Civil  Military  Operations  Task  Force  (CJCMOTF) 
team.  The  team  is  responsible  for  monitoring  and  coordinating  the  activities  of  Provincial 
Reconstruction  Teams  (PRTs),  as  well  as  International  Organizations  and  NGOs.  High  Level 
Focus  Areas  broached  by  this  vignette  include:  identifying  capabilities  and  limitations  of  your 
coalition  partners;  understanding  your  coalition  partners’  motivation,  objectives,  reason  for 
participating  in  mission;  adaptability  of  skill  sets;  organizational  structure;  information 
exchange;  integrating  resources  to  accomplish  the  mission;  understanding  the  intent  of  other 
coalition  commanders;  approach  to  problem  solving;  awareness  and  implications  of  actions, 
understanding  NGO’s  motivations,  goals  and  agendas;  force  protection;  and  ROE,  In  Situation 
1,  a  large  amount  of  money  is  available  for  distribution  across  the  country  for  reconstruction. 
Players  must  describe  ways  to  promote  greater  coordination  between  the  PRTs,  and  identify 
failures  in  coordination  which  have  taken  place.  In  Situation  2,  the  World  Health  Organization, 
the  Red  Cross,  and  coalition  partners  are  coordinating  a  medical  distribution  site.  Players  are 
required  to  identify  critical  issues  in  collaborations  with  NGOs  and  determine  why  the  U.S. 
military  and  NGO  coalition  partners’  goals  may  conflict.  In  Situation  3,  players  are  members  of 
the  HQ  Operational  Plans  Group,  responsible  for  developing  a  plan  to  manage  security 
throughout  the  country  so  that  the  overall  mission  of  security  and  nation  building  can  be  realized. 
Players  must  coordinate  the  activities  of  coalition  partners  and  identify  the  priorities  and 
appropriate  posture  of  security  forces. 

Phase  3:  Developing  the  Learning  Strategy’  and  Training  Method 

In  this  report,  we  use  the  term  training  method  to  refer  to  the  training  platform  or  tool 
itself;  that  is,  the  method  or  mechanism  for  delivering  training.  In  this  case,  the  training  method 
is  the  use  of  a  low-fidelity,  synchronous,  web-based,  distributed  training  tool.  The  learning 
strategy  refers  to  how  the  training  content  is  structured  and  facilitated,  the  expectations  and  goals 
we  have  for  learning,  and  the  rationale  for  those  factors. 
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Learning  Strategy  and  Facilitation  Guides 

The  learning  strategy  employed  in  this  project  is  designed  to  support  participants'  mental 
model  development  regarding  coalition  challenges  and  how  the  challenges  can  he  resolved  to 
greater  effectiveness  in  mission-specific  circumstances.  As  noted  above,  the  themes  we  found 
consistently  in  the  experiences  of  our  interv  iewee  participants  provide  the  challenges  (as  well  as 
opportunities  for  success)  for  officers  in  coalition  operation.  These  themes  were  incorporated 
into  the  scenarios.  To  build  mental  models  about  how  coalitions  can  collaborate,  the  officers 
must  accumulate  experiences  or  surrogate  experiences  working  through  cognitive  challenges. 
Surrogate  experiences,  facilitated  to  help  the  officers  reflect  on  their  assumptions  and 
performance,  is  the  basis  of  the  learning  strategy.  Barriers  to  coordination  effectiveness  in  the 
coalition  environment  (other  than  the  ubiquitous  discussions  of  technology  interoperability) 
include  areas  of  expertise  and  capability  and  differences  in  assumptions  about  goals  and  how  to 
achieve  them.  As  awareness  and  understanding  are  developed,  mental  models  of  collaboration 
will  develop.  The  mental  models  will  include  expectations  for  different  situations  and  different 
allied  forces,  the  nature  of  goals,  and  action  plans  and  cues  that  indicate  whether  collaboration  is 
proceeding  successfully  or  not.  Thus  the  basis  for  the  initial  learning  strategy  was  1 )  to  build 
expertise  in  the  decisions  that  need  to  be  made  in  coalition  operations  through  practice  in 
working  through  challenges  in  the  context  of  scenarios,  and  2)  to  build  mental  models  that 
support  that  expertise  by  having  participants  recognize  the  importance  and  meaning  of  aspects  of 
staff  collaboration  within  specific  theme  areas — our  HLFAs. 

Facilitation  of  the  training  session  is  key  to  the  teaming  strategy.  Successful  facilitation 
strategies  include:  1 )  engaging  in  deliberate  practice,  so  that  each  opportunity  for  practice  has  a 
goal  and  evaluation  criteria;  2)  obtaining  feedback  that  is  accurate  and  diagnostic;  3)  building 
mental  models;  4)  developing  metacognitive  skills;  and  5)  becoming  more  mindful  of 
opportunities  for  learning  (Pliske,  McCloskey,  &  Klein.  2001).  Elaboration  and  comparison  of 
each  person's  decision  making  process  brings  out  the  assumptions,  cues  and  factors,  challenges, 
and  strategies  of  everyone  for  comparison.  These  elements  generally  comprise  the  mental  model 
and  situational  understanding  of  the  training  participants,  and  the  process  allows  each  person  to 
develop  more  insight  into  situations,  thus  developing  their  own  mental  models. 

The  role  of  the  facilitator  is  an  important  topic  in  education  today.  Many  different  types 
of  content  are  now  deli  vered  through  web-based  learning.  Many  of  these  training  courses 
require  facilitation  because  they  involve  concepts  that  cannot  easily  be  represented  and  processed 
by  the  student  without  facilitation.  Collaborative  learning  is  also  a  growing  area  of  research  and 
development.  To  many  educators  and  researchers,  the  use  of  technology  is  secondary  to  the 
development  of  learning  goals,  content,  and  process.  Rather,  learning  needs  to  be  facilitated  for 
meaningful  responses  and  insights  to  occur  regardless  of  the  technology  (Berge.  1995). 

However,  the  training  method  or  platform  can  support  or  hinder  the  processes  as  well  as  match 
or  impede  the  needs  of  the  learning  audience.  As  we  developed  our  learning  strategy  we 
attempted  to  synchronize  the  facilitation  process,  the  content,  and  the  technology  supporting  the 
facilitation  process. 

A  facilitation  guide  supports  each  of  the  three  scenarios.  Each  guide  includes:  the 
teaching  points  targeted  in  the  scenario,  the  requirements  set  for  the  students,  and  a  process  for 
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supporting  the  facilitator  in  leading  the  discussion.  Potential  student  questions  are  also  included, 
some  with  answers  provided  by  SMEs.  The  facilitator  must  choose  what  questions  to  take,  and 
when  to  say  ‘"that  information  is  not  available.”  The  guides  for  each  scenario  are  integrated  into 
the  web-based  tool  to  access  online  before  and  during  training.  The  facilitation  guides  are  not 
meant  to  fully  train  an  instructor/facilitator  in  the  art  of  facilitation;  rather,  experienced 
facilitators  are  needed  to  capitalize  on  the  training.  (Full  versions  of  the  facilitation  guides  are 
available  in  the  accompanying  CD  attachment  entitled  Global  Teams  Scenarios  and  Facilitation 
Guides.  They  are  also  available  online  at  http://www.361interactive.com/globalteams/index. 
html). 

Training  Me  (hod -The  Global  Teams  Collaborative  Training  Tool 

The  Global  Teams  tool  is  a  collaborative,  synchronous  training  tool  developed  to  support 
representatives  of  military  and  non-military  organizations  of  different  countries  in  enhancing 
mutual  understanding  and  communication  skills.  The  tool  allows  facilitators  to  upload  and 
conduct  training  vignettes  with  players  distributed  across  the  world,  and  can  support  up  to  25 
concurrent  users.  Features  such  as  shared  drawing  spaces,  shared  and  private  chat  and  response 
areas,  and  optional  streaming  audio  support  provide  a  truly  interactive  environment  where 
facilitators  can  lead  players  through  an  engaging,  interactive  learning  experience. 

The  functionality  of  the  tool  and  the  interface  design  support  the  learning  strategy,  not 
any  particular  HLFA  or  AP.  The  tool  was  designed  to  be  flexible  and  support  a  range  of  training 
options  that  depend  on  the  strategy  of  facilitated,  surrogate  experiences.  The  tool  does  allow  the 
users  to  address  the  HLFAs  for  each  scenario,  and  supports  both  collaboration  and  facilitation, 
key  elements  of  the  strategy.  How  well  the  tool  supports  the  HLFAs  and  APs  is  highly 
dependent  on  the  ability  of  the  facilitator.  Descriptions  of  the  functionalities  and  interface  are 
presented  below.  Of  those  functionalities,  the  ability  to  manipulate  graphics  on  the  map  allows 
for  collaboration  by  the  players  as  well  as  the  ability  of  the  facilitator  to  highlight  information. 
The  shared  whitespace  allows  synchronous  sketching  of  such  things  as  initial  concepts  of 
operation  or  even  assumptions  about  the  area  of  operations.  Private  response  forms  provide  the 
ability  for  the  facilitator  to  gain  an  understanding  of  the  different  points  of  view  and  assumptions 
of  the  player  participants  prior  to  bringing  them  together  to  share  those  views.  As  the  desired 
outcome  is  for  players  to  develop  their  mental  models  around  the  HLFAs,  this  feature  was 
designed  to  allow  opportunities  for  the  facilitator  to  assess  changes  over  time  in  responses  in 
both  formative  evaluations  in  real  time  and  summative  evaluations  after  training.  The  Split- 
Screen  allows  the  facilitator  to  bring  individually  developed  points  of  view  together  for  all 
participants  to  compare  to  further  support  insight  into  how  collaborations  may  be  approached  or 
even  impeded.  The  chat  features  allow  users  to  hold  group  discussions,  and  for  researchers  and 
facilitators  to  exchange  views  in  real  time  as  well  as  to  direct  the  sessions. 

The  training  method  is  purposely  designed  to  be  low  tech  to  allow  access  to  the  range  of 
potential  users  who  may  have  low  levels  of  technology  to  support  their  training.  The  simple 
technology  design  attempts  to  minimize  technology  problems  for  a  variety  of  users  accessing  the 
web-based  tool.  In  addition,  a  goal  for  the  tool  was  to  support  future  researchers  in  their  ability 
to  easily  add  scenarios  for  continued  or  other  research  purposes. 
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The  training  platform  is  an  ideal  solution  for  the  audience  that  is  geographically 
separated.  The  simplicity  of  the  method  allows  ad  hoc  teams  to  form  easily,  and  without  much 
preparation,  to  conduct  training  (if  the  facilitator  is  conversant  in  the  learning  strategy  and  the 
scenario  forming  the  focus  of  the  training).  All  the  tools  needed  are  present  online  and  relieve 
the  training  audience  from  acquiring,  loading  software,  carrying  training  products  with  them,  and 
being  in  any  certain  location  to  train. 

The  research  team  established  the  desired  functionality  for  a  training  platform  that  would 
support  web-based,  synchronously  facilitated  training  sessions,  including  some  functions  that 
were  desired  but  not  necessarily  required  in  this  effort,  such  as  voice  over  internet  protocol.  The 
requirements  at  this  point  were  based  on  existing  training  products,  tools  available  for  web-based 
collaboration  in  a  low  technology  format,  and  the  need  to  support  scenario-based  training.  We 
continued  development  of  the  basic  design  through  iterative  sessions.  The  primary  supported 
functionalities  are  presented  in  Appendix  A. 

Training  Interface 

Figure  1  presents  a  screen  capture  of  the  Global  Teams  main  page,  to  which  all  logged-in 
users  will  be  directed  after  selecting  a  vignette  from  the  library,  A  majority  of  this  screen  is 
common  to  all  users.  However,  only  registered  facilitators  have  full  access  to  the  facilitator  chat 
area  and  the  facilitator  guidance.  Also,  only  facilitators  can  view  each  team’s  response  forms  at 
any  time.  The  functional  areas  of  the  main  page  are  as  follows: 

Shared  Workspace  Area.  The  largest  area  of  the  main  page  contains  the  shared 
workspace.  Here,  users  can  view  and  mark  up  the  maps  and  graphics  associated  with  a  selected 
vignette.  By  default,  the  vignette’s  background  graphic  is  displayed  initially.  However,  users 
can  alternatively  navigate  to  any  of  the  vignette’s  graphics  (background.  Situation  1 ,  Situation  2, 
Situation  3),  a  shared  whiteboard  area,  or  the  shared  space  text  area  at  any  time  by  clicking  on 
the  tabs  on  the  top  of  the  workspace.  The  currently  selected  graphic/workspace  is  designated  by 
the  tab  color  changing  from  black  to  red.  Along  the  left  side  of  the  shared  workspace  is  the 
workspace  toolbar.  Using  these  individual  toolbar  buttons,  users  can  superimpose  textboxes, 
free  text,  arrows,  and  a  range  of  shapes  onto  the  graphic  for  all  other  users  to  see.  Users  can  also 
draw  straight  or  freeform  lines,  can  customize  the  colors  of  their  additions,  and  can  selectively 
move/delete  individual  items  or  clear  the  entire  workspace.  The  toolbar  can  be  hidden  and  re¬ 
shown  by  clicking  the  small  blue  arrow.  The  background  graphic  is  displayed  initially. 

However,  users  can  alternatively  navigate  to  any  of  the  vignette's  graphics  (at  the  bottom  of  the 
bar).  The  shared  space  text  area  presents  a  dual  text  region  where  users  can  input  and/or 
compare  multiple  answers  simultaneously  as  a  group. 

Username  Login/Control  Area.  At  the  top  right  comer  of  the  main  screen,  users  can 
change  their  onscreen  names  by  entering  them  in  the  textbox  and  clicking  the  Login  button.  This 
feature  is  especially  useful  if  multiple  parties  are  logged  in  as  facilitators.  Each  facilitator  can 
elect  a  unique  name  and  that  name  will  be  associated  with  their  cursors,  audio  messaging,  and 
group  chats. 
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Figure  1 .  Global  Teams  main  page. 

Group  Chat/Response  Form  Area.  Directly  below  the  login  area  is  the  group 
chat/response  form  area.  By  default,  when  the  user  arrives  at  this  main  page,  the  group  chat 
functionality  is  enabled.  Here,  all  users  can  freely  communicate  via  instant  text  chatting.  Text  is 
entered  by  an  individual  user,  and,  upon  clicking  the  Send  button,  the  user’s  text  is  displayed  for 
all  other  users  to  instantly  view.  That  user's  text  is  uniquely  identified  by  both  username  and 
color  (if  that  user  has  selected  a  unique  color+see  control  area.  To  access  the  group  chat,  the 
user  can  click  on  the  group  chat  tab  at  any  time.  To  clear  the  chat  window,  a  user  can  click  on 
the  clear  chat  button  beneath  the  chat  window.  The  second  tab  displays  individual  player 
response  forms.  When  a  registered  player  clicks  on  the  response  form  tab,  the  group  chat  will  be 
replaced  onscreen  by  a  text-based  response  form.  Here,  the  player  can  enter  responses  to 
questions  presented  by  the  facilitator.  While  the  other  players  cannot  view  this  information, 
registered  facilitators  can  view  it  at  any  time.  When  a  facilitator  clicks  on  the  response  form  tab, 
the  USA  response  form  appears  by  default.  In  addition,  five  buttons  (representing  the  countries 
of  USA,  UK,  Canada,  and  Australia,  and  a  generic  non-government  organization)  appear  below 
the  form.  These  buttons  are  only  visible  to  facilitators.  By  clicking  on  these  buttons,  the 
facilitator  can  observe  the  progress  of  each  individual  player  in  answering  the  questions.  A  help 
button  in  this  area  generates  a  new  popup  window  with  guidance  in  using  these  features  as  well 
as  the  name  changing  feature. 

Control  Area.  The  bottom,  right  comer  of  the  main  page  contains  the  control  area. 
Several  functionalities  are  located  here.  The  box  beneath  the  audio  label  displays  all  currently 
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logged-in  users  by  username,  It  also  serves  a  second,  optional  purpose.  By  pressing  the  talk 
button,  users  with  microphones  can  speak  to  the  other  distributed  users  (if  they  have  functional 
speakers).  The  user  can  either  press  and  hold  down  the  talk  button  to  speak,  ensuring  privacy 
when  they  don’t  want  conversations  overheard,  or  they  can  click  the  auto  box,  and  the  tool  will 
detect  when  the  user  is  speaking  automatically.  Lights  to  the  left  of  each  username  in  the  box 
flash  green  when  that  particular  user  is  speaking. 

The  facilitator  chat  box  to  the  right  of  the  audio  box  is  a  private  area,  only 
accessible/visible  to  registered  facilitators,  where  such  users  can  conduct  administrative  chats 
that  would  otherwise  distract  players. 

The  pointer  color  selector  beneath  the  facilitator  chat  box  allows  any  user  to  selectively 
customize  his  or  her  cursor  and  group  chat  color.  This  feature  supports  the  distinction  of 
distributed  users. 

The  buttons  at  the  bottom  of  the  control  area  generate  new  windows  that  access  Word- 
formatted  text  documents.  These  documents  provide  the  relevant  rules  of  engagement, 
background  and  situational  information  associated  with  the  selected  vignette.  By  clicking  on  a 
button,  the  user's  browser  will  open  a  new  window  for  the  text-based  information,  allowing  the 
user  to  simultaneously  view  the  main  page/graphics  and  supporting  text.  The  facilitator  too! 
button  is  only  visible  to  registered  facilitators,  and  provides  supporting  documentation  for 
conducting  vignettes.  A  help  button,  visible  to  all  users,  generates  a  separate  popup  window 
with  additional  guidance  on  using  the  control  area. 

Phase  4:  Training  Evaluation 

Training  evaluation  took  place  in  two  parts:  a  preliminary  evaluation  and  a  main 
evaluation.  The  purpose  of  the  preliminary  evaluation  was  to  test  the  functionality  of  the  training 
tool  and  the  usability  of  the  training  strategy.  The  main  evaluation  was  designed  to  test  the 
effectiveness  of  the  collaborative  distributed  web- based  training  methodology  for  supporting 
Soldiers'  understanding  of  coalition  operational  challenges.  Four  levels  of  evaluation  were  used 
to  accomplish  this  goal.  The  measures  are  described  below. 

Training  evaluation  was  based  on  Kirkpatrick’s  (1976)  hierarchy  for  training  evaluation. 
The  hierarchy  consists  of  four  different  levels  of  evaluation:  reactions.  learning,  behavior,  and 
organization.  These  levels  represent  a  sequence  of  methods  to  evaluate  a  training  program. 

Each  level  builds  on  the  previous  one,  but  also  provides  more  valuable  information. 

Level  1:  Reactions:  Survey  and  Discussion 

Reactions  are  concerned  with  how  the  participants  react  to  the  training.  It  is  important  to 
note  that  positive  reaction  does  not  ensure  learning,  although  a  negative  reaction  almost  certainly 
reduces  the  likelihood  that  learning  has  taken  place  (Kirkpatrick,  1998).  To  learn  about 
participants'  reactions  to  the  training,  we  developed:  1)  a  short  survey  to  be  filled  out  by  each 
participant  at  the  end  of  each  training  day  (See  Appendix  B  for  Reactions  Survey)  and  2)  a  set  of 
discussion  questions  to  guide  the  elicitation  of  the  participants’  reactions  (See  Appendix  C  for 
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Discussion  Questions).  This  discussion  took  place  in  an  open  forum  (or  a  focus  group)  at  the  end 
of  the  training  session.  The  idea  was  to  initiate  a  discussion  about  participants’  perceptions  of 
the  usefulness,  value,  and  relevance  of  the  training. 

Level  2:  Learning:  Situation  Awareness  (SA)  Exercise 

Learning  is  the  second  level  in  the  hierarchy,  and  refers  to  "the  principles,  facts,  and 
skills  which  were  understood  and  absorbed  by  the  participants"  (Kirkpatrick,  1976,  p.  11).  This 
level  is  concerned  with  whether  the  participants  have  acquired  knowledge,  or  have  modified  their 
attitudes  or  beliefs  as  a  result  of  attending  the  training  course.  It  is  important  to  measure 
learning,  as  no  change  in  behavior  can  be  expected  if  no  new  knowledge  or  change  in  attitudes 
has  occurred.  In  this  case,  learning  refers  to  whether  participants  have  more  knowledge  about 
coalition  operations  after  the  training.  In  order  to  test  the  learning,  three  SA  Exercise  questions 
(Pliske,  Militello.  Phillips,  &  Battaglia,  2001 )  were  embedded  into  the  training  tool.  These  three 
questions  were  asked  after  each  of  the  three  vignettes  was  presented. 

Level  3:  Behavior:  Behavioral  Markers 

An  evaluation  at  the  behavior  level  is  the  assessment  of  whether  knowledge  learned  in 
training  actually  transfers  to  behaviors  on  the  job  or  a  similar  simulated  environment.  For  each 
of  our  scenarios,  three  separate  observations  were  made,  one  for  each  vignette.  Observ  ations 
were  made  using  a  behavioral  marker  system.  Behavioral  markers  are  descriptions  of 
observable,  non-technical  behaviors  that  contribute  to  superior  or  substandard  performance 
within  a  work  environment.  They  describe  specific,  observable  behaviors — not  attitudes  or 
personality  traits — with  clear  definition  (enactment  of  skills  or  knowledge  is  shown  in  behavior). 
Each  behavior  should  have  a  demonstrated  causal  relationship  to  performance  outcome.  One  of 
the  main  purposes  of  behavioral  marker  systems  is  to  enable  performance  measurement  for 
training  evaluation.  According  to  standard  practice,  observers  made  notes  while  observing, 
noting  examples  of  positive  and  negative  behaviors  based  on  the  dimensions  highlighted  in  the 
system.  At  the  end  of  the  session  the  raters  rated  the  team  on  each  dimension. 

The  behavioral  marker  system  we  used  is  adapted  from  the  NOTECHS  system 
(Avermaete  &  Kruijsen,  1998).  NOTECHS  was  developed  by  EC  DGTREN  (European 
Community  Directorate  for  Transport  and  the  Environment)  and  the  Civil  Aviation  Authorities 
of  France,  Netherlands,  Germany,  and  UK.  The  system  was  developed  as  a  method  for 
instructors  to  assess  pilots’  non-technical  skills  during  training.  The  system  consists  of  a 
framework  of  desirable  behaviors  in  a  hierarchy  of  categories  and  elements.  In  the  system  we 
used,  there  are  five  primary  categories  and  18  elements  as  shown  in  Table  2.  The  categorical  set 
was  based  on  theoretical  models  identified  from  the  team  working  literature  (Cannon-Bowers  & 
Salas,  1997;  Zsambok.  Klein,  Kyne,  &  Klinger,  1993).  Evaluators  are  required  to  indicate 
whether  a  particular  element  of  behavior  was  observed  during  training  and  then  make  a  judgment 
about  the  team's  performance  on  each  category. 
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Tabic  2 


Framework  of  Desirable  Behaviors 


Category 

Description 

Element 

Example 

Team  identity 

The  extent  to 
which  members 
conceive  of  the 
team  as  an 
interdependent  unit 
and  then  operate 
from  that 

perspective.  When 
members  have 
weak  team  identity 
they  tend  to 
operate  as 
individuals  instead 
of  as  members  of  a 
team. 

Defining  roles  and 
functions 

•  Are  the  team  members  actively 
assigning  roles  and 
responsibilities? 

•  Are  they  updating  roles  and 
responsibilities  as  the  situation 
changes? 

Engaging 

•  Is  the  team  capitalizing  on  team 
member  participation? 

•  Has  any  member  of  the  team 
disengaged?  Signs  that  may 
indicate  a  team  member  has 
disengaged: 

•  Not  paying  attention  during 
discussions. 

•  Performing  a  different  task  during 
the  discussion. 

•  Not  following  up  on  questions  or 
concerns. 

Compensating 

•  Have  members  of  the  team 
stepped  outside  their  assigned 
roles  or  functions  in  order  to  help 
the  team  achieve  its  goals? 

Avoiding  micro- 
management 

•  Are  people,  tasks,  or  information 
being  managed  at  the  wrong  level 
of  detail? 

Team  self 
monitoring 

Ability  of  the  team 
to  observe  and 
monitor  itself. 

Time 

management/ 

workload 

management 

•  Are  the  team  members  managing 
workload  correctly? 

•  Are  they  aware  of  the  time 
limitation  on  tasks  and  planning 
work  accordingly? 

Adjusting 

•  Is  the  team  recognizing  and 
adjusting  for  problems? 

(T able  Continues) 
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Category 

Description 

Element 

Example 

Cooperation 

The  sense  of 
‘team'  within  the 
team  and  their 
willingness  to 
support  and  help 
each  other. 

Teambuilding  and 
maintaining 

•  Is  there  an  atmosphere  for  open 
communication  and  participation? 

•  Is  there  input  and  feedback  from 
everyone  on  the  team? 

Consideration  of 
others 

•  Do  the  team  members  take  notice 
of  the  suggestions  of  all  members 
even  if  they  do  not  agree? 

Support  of  others 

•  Do  team  members  help  each  other 
in  demanding  situations? 

Conflict  solving 

•  Do  the  team  members  concentrate 
on  what  is  right  rather  than  who  is 
right? 

Team 

conceptual  level 

A  high  team 
conceptual  level  is 
achieved  through 
the  maintaining  of 
common  ground, 
and  a  shared 
understanding  of 
goals,  objectives, 
and  situation 
assessment. 

Envisioning  goals 
and  processes 

•  Have  the  team  members  dearly 
communicated  their  goals? 

•  Have  they  clearly  communicated 
the  plan? 

•  Have  team  members  asked  for 
further  clarification? 

Forecasting:  time 
horizons  and 
range  of  factors 

•  Have  the  team  members 
demonstrated  the  ability  to  focus 
decision  making  within  an 
appropriate  span  of  time? 

•  Have  the  team  members 
demonstrated  the  ability  to  focus 
on  a  relevant  breadth  of  concepts 
and  information? 

•  I  lave  the  team  members 
demonstrated  the  ability  to  project 
the  results  of  their  actions  into  the 
future? 

Detecting  gaps 
and  ambiguities 

•  Have  the  team  members  been  able 
to  easily  recognize  when  they  are 
missing  certain  information  and 
when  assumptions  are  inaccurate? 

•  Have  they  had  difficulty  in 
determining  that  information  is 
missing? 

•  Have  they  tended  to  ignore 
ambiguity? 

(T able  Continues) 
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Category 

Description 

Element 

Example 

Team 

conceptual  level 

continued 

Achieving 

situation 

assessment: 
diverging  and 
converging 

•  Has  the  team  sought  alternate 
explanations  and  assessments  from 
team  members? 

•  Has  the  team  reached  a  shared 
understanding  of  the  situation? 

•  Has  the  team  employed  mental 
simulation  and  perspective  taking? 

Team  Decision 
Making 

The  extent  to 
which  decision 
making  is  a 
sequential  and 
deliberative 
process.  In  high 
performing  teams 
diagnosis  precedes 
option  generation, 
risk  assessment, 
and  finally  review. 

Problem 
definition  and 

diagnosis 

•  Has  the  team  actively  gathered 
information  and  identified 
problems? 

•  Have  they  actively  reviewed  causal 
factors? 

Option  generation 

•  Has  the  team  stated  an  alternative 
Course  of  Action  (CoA)? 

•  Have  they  asked  members  for 
options? 

Risk  assessment 
and  option 
selection 

•  Has  the  team  considered  the  risks 
of  alternative  CoAs? 

•  Have  they  talked  about  possible 
risks  for  CoA? 

•  Has  the  team  confirmed  the  CoA 
with  all  members? 

Outcome  review 

•  Has  the  team  checked  the  outcome 
against  the  plan? 

The  system  also  includes  an  elements  score  form  on  which  the  observer  notes  whether  the 
behavior  was  observed  or  not  and  a  score  form  on  which  the  observer  rates  how  well  the  team 
performs  overall  on  that  category.  The  evaluators  first  indicated  whether  a  particular  element  of 
behavior  was  observed.  Then,  following  the  end  of  the  exercise,  the  evaluator  made  a  judgment 
about  the  team's  performance  on  each  of  the  five  dimensions  (See  Appendix  D  for  elements  and 
categories  score  forms). 

Level  4:  Organization 

This  is  the  highest  level  of  evaluation  in  Kirkpatrick’s  (1976)  hierarchy.  The  ultimate 
aim  of  any  training  program  is  to  produce  tangible  evidence  at  an  organizational  level.  The 
problems  with  evaluating  training  at  this  level  are  that  it  can  be  difficult  to  establish  discernible 
indicators  and  to  be  able  to  attribute  these  to  the  effects  of  a  single  training  course.  For  obvious 
reasons,  it  is  too  early  to  try  to  measure  organizational  level  change  directly  following  the 
training  exercise.  However,  we  hope  to  be  able  to  infer  whether  there  is  likely  to  be 
organizational  level  change  from  a  comparison  of  the  behavioral  analysis  across  the  three 
vignettes  in  each  scenario.  If  we  see  consistent  change  in  the  right  direction  across  the  three 
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vignettes,  over  three  scenarios,  we  will  be  able  to  inter  with  some  level  of  certainty  that 
participants  are  accommodating  different  points  of  view  as  a  result  of  this  training,  I  lowever, 
within  the  limits  of  this  project  it  was  not  possible  to  develop  any  specific  measure  of 
Organizational  level  change. 


Results  of  the  Training  Evaluation 
Preliminary  Evaluation 

Two  assessment  methods  were  used  in  the  preliminary  evaluation.  First  the  reactions 
survey  was  used  to  determine  how  participants  reacted  to  the  training.  Second,  the  post  training 
discussion  was  used  to  get  participants'  feedback  on  the  training  and  the  tool.  The  outcome  of 
this  assessment  process  was  used  to  guide  our  development  process.  Preliminary  evaluations 
took  place  at  the  Joint  Readiness  Training  Center  (JRTC)  at  Fort  Polk,  Louisiana.  Three  U.S. 
Army  JRTC  Observer/Controllers  (O/Cs)  took  part  in  the  preliminary  evaluation.  There  were 
two  Captains  (CPT)  and  one  LTC  who  acted  as  the  U.S.  training  participants.  Two  of  the  three 
participants  had  previous  coalition  operational  experience  in  Iraq.  An  SME  (retired  Army 
officer)  with  coalition  experience  played  the  roles  of  the  Australian.  Canadian,  and  UK 
contingent.  The  purpose  of  the  preliminary  evaluation  was  not  to  test  the  learning  taking  place, 
but  rather  to  test  the  functionality  of  the  training  tool  and  the  usability  of  the  training  strategy. 
Thus,  preliminary  evaluation  data  were  limited  to  the  Reactions  Survey  and  Discussion 
Questions  -  as  discussed  in  the  training  evaluation  plan. 

Preliminary  Evaluation  Reactions  Survey  Results 

Participants  were  asked  to  rate  on  a  scale  of  1  {none  at  all)  to  5  (a  great  deal)  their 
answers  to  four  questions  about  the  training  they  had  just  been  through.  In  all  cases,  the  number 
of  participants  was  three.  Mean  ratings  are  provided  here  as  the  consistency  across  participants' 
responses  was  high. 

•  How  much  value  is  there  to  training  with  coalition  partners  in  general?  Mean  rating  =  4.6 

•  Howr  much  value  was  there  in  training  with  coalition  partners  in  this  exercise?  Mean 
rating  =  2 

•  How  would  you  rate  the  quality  of  the  content  presented  in  this  exercise?  Mean  rating  = 
2.33 

•  How  frequently  should  a  staff  member  participate  in  similar  training  (On  a  scale  of  1 
(never)  to  5  (once  a  month)?  Mean  rating  =  2.6 

Participants  were  also  asked  to  list  three  things  that  made  the  training  effective.  All  three 
participants  stated  that  interaction  with  partners,  working  together  to  solve  a  problem,  and  the 
conversation  with  partnering  nations  were  the  factors  that  made  the  training  effective.  When 
asked  to  list  the  three  things  that  would  make  the  training  more  effective,  the  following  are  the 
responses  in  order  of  the  number  of  times  they  were  mentioned. 

•  Greater  use  of  graphic  tools. 

•  More  detailed  maps,  more  detailed  information  in  the  scenarios. 
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•  More  incorporation  of  doctrine. 

•  The  ability  to  interact  to  solve  problems. 

A  final  question  asked  participants  to  provide  any  additional  comments  they  would  like 
to  make  about  the  training.  Comments  included:  1 )  The  need  for  greater  clarity  about  the 
objectives  of  the  exercise  before  the  training;  and  2)  the  need  to  set  the  dilemmas  at  a  level 
appropriate  to  the  rank  and  role  of  the  participants  and  the  role  they  are  being  asked  to  play  in  the 
scenario.  For  example,  battalion  (BN)  and  brigade  (BDE)  operational  level  Multi-National 
Division  (MND)  staffs  deal  with  tactical  issues  while  CJTF  staffs  deal  with  strategic,  political, 
and  long  term  issues. 

Preliminary  Evaluation  Discussion  Results 

Participants  provided  feedback  on  four  main  issues:  the  tool  itself,  the  scenarios,  the 
participant  requirements,  and  the  facilitation.  A  summary  of  the  points  they  raised  is  presented 
below. 


The  three  participants  felt  that  that  there  was  a  lack  of  connection  between  the  graphic 
and  the  text;  they  would  have  liked  more  opportunity  to  put  the  graphic  to  use  in  solving  the 
problem.  In  fact,  the  requirements  (or  challenges  set  for  participants)  were  seen  as  not  specific 
enough  to  require  the  use  of  the  map.  A  related  issue  is  the  lack  of  a  cause  and  effect  dilemma 
with  which  participants  could  grapple.  There  was  also  consensus  on  the  difficulties  involved  in 
using  the  chat  function  as  a  main  means  of  communication.  The  “chat"  became  disjointed  and 
difficult  to  follow;  it  was  generally  agreed  that  using  the  speaker  phone  was  more  useful  than  the 
chat  function. 

The  participants  saw  the  scenarios  as  realistic  and  accurate,  but  needing  to  be  pitched  at 
the  appropriate  level  (i.e.,  at  the  tactical  level  for  BN  and  BDE  level  staffs,  and  at  the  strategic 
level  for  CJTF  level  staffs  [these  would  be  theater  level  issues]).  They  felt  the  scenarios  should 
provide  more  information  about  capabilities,  and  the  process  should  include  giving  different 
information  to  each  team.  Finally,  the  facilitation  should  provide  the  opportunity  for  more 
coordination  during  problem  solving,  and  learning  objectives  should  be  given  up  front. 

In  response  to  the  issues  raised  by  participants  through  the  pre-evaluation  reaction  survey 
and  discussion  session,  modifications  were  made  to  the  scenarios,  the  participant  questions  and 
the  facilitation  guides.  For  example,  participant  questions  were  modified  to  allow  for  greater 
interactions  and  discussion  over  problem  diagnosis  as  well  as  problem  solving.  The  factual 
details  in  the  scenarios  were  changed  to  make  them  more  level  appropriate,  and  potential 
participant  questions  and  answers  were  modified  to  reflect  the  changes. 

Main  Evaluation 

A  total  of  18  participants  took  part  in  the  main  evaluation  (nine  from  the  U.S.  Army, 
three  from  the  Canadian  Army,  three  from  the  UK  Army,  and  three  from  the  Australian  Army). 
Of  the  1 8  participants,  the  ranks  included  five  CPTs,  four  LTCs,  two  commanders  (CMDRs), 
and  seven  MAJs.  Fourteen  participants  had  previous  experience  working  with  coalition  partners. 
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In  total,  nine  had  coalition  experience  in  Iraq,  six  had  coalition  experience  in  Bosnia,  two  had 
experience  working  with  coalition  partners  in  Germany,  and  one  in  East  Timor.  Four 
participants  had  no  experience  working  with  coalition  partners.  Most  of  the  participants  held 
positions  related  to  training,  development,  and  operational  analysis  including:  doctrine  writer; 
exercise  planner;  operational  training  developer;  distributed  learning  director;  leadership  training 
planner;  deputy  director  of  training  and  development;  program  manager  for  training,  doctrine 
and  combat  development;  course  tutors. 

Evaluation  took  place  over  three  days.  On  each  day  three  U.S.  players  and  three  players 
from  either  Australia,  Canada,  or  the  UK  played  through  one  scenario  consisting  of  three 
vignettes.  The  teams  were  connected  via  telephone  and  through  the  web-based  training  tool.  An 
observer  accompanied  each  team.  The  observer  provided  participants  with  hard  copies  of  the 
vignettes  and  distributed  the  survey  at  the  end  of  the  sessions.  A  remote  facilitator  experienced 
in  military  coalition  operation  facilitated  the  training  sessions.  Three  training  developers  and  one 
tool  developer  remotely  observed  the  training  and  facilitated  the  survey  and  discussion  sessions. 

Level  1:  Reactions:  Survey  and  Discussion 

Level  1  refers  to  how  participants  reacted  to  the  training.  While  negative  and  positive 
reactions  are  not  necessarily  indicators  of  learning,  the  reactions  were  used  to  guide  our 
development  process. 

Survey  Results.  Participants  rated  on  a  scale  of  1  (none  at  all)  to  5  (a  great  deal)  the 
following  questions  about  the  training  they  had  just  been  through.  In  all  cases,  the  number  of 
participants  was  18. 

•  How  much  value  is  there  to  training  with  coalition  partners  in  general?  Mean  rating  =  4.9 

•  How  much  value  was  there  in  training  with  coalition  partners  in  this  exercise?  Mean 
rating  =  4 

•  How  would  you  rate  the  quality  of  the  content  presented  in  this  exercise?  Mean  rating  = 
3.3 

•  How  frequently  should  a  staff  member  participate  in  similar  training?  On  a  scale  of  1 
(never)  to  5  (once  a  month)  Mean  rating  =  3.7 

When  asked  to  list  three  things  that  made  the  training  effective,  three  main  factors  were 
identified  as  key  to  the  effectiveness  of  the  training  (responses  are  listed  in  order  of  frequency  of 
mention  by  participants): 

•  Conversation  with  partnering  nations. 

•  Realistic  scenarios. 

•  Experienced  facilitator. 

All  of  the  respondents  agreed  that  conversation  between  the  partners  was  the  most 
effective  aspect  of  the  training.  This  provided  them  with  the  opportunity  to  exchange  views, 
discuss  doctrinal  differences,  and  generate  greater  awareness  of  similarities  and  differences  in 
operational  activities.  The  fact  that  this  discussion  was  facilitated  by  an  experienced  facilitator. 
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and  that  it  was  in  connection  with  relevant  operational  issues  was  also  acknowledged  by  a 
majority  as  being  highly  effective.  It  was  acknowledged  that  the  experienced  facilitator  provided 
probing  questions  and  extra  information  that  facilitated  a  broadening  of  perspective.  A  majority 
of  respondents  said  that  the  scenarios  were  realistic,  authentic,  and  factually  accurate. 

Respondents  were  also  asked  to  list  three  things  that  would  make  the  training  more 
effective.  Below  are  the  responses  listed  in  the  order  of  frequency  that  they  were  mentioned  by 
participants.  Six  factors  were  identified: 

•  Better  use  of  graphics  and  graphic  tools. 

•  Greater  interactivity. 

•  Read-ahead  materials. 

•  More  time  to  consider  the  problems. 

•  Tailor  the  exercises  to  the  target  audience  and  pitch  requirements  at  the  correct  level. 

•  Greater  closeness  to  Military  Decision-Making  Process  (MDMPj  and  military  doctrine. 

There  was  general  consensus  regarding  the  need  for  greater  interactivity  between  the 
requirements  set  for  participants  and  the  graphic.  In  total  five  participants  would  have  preferred 
having  a  requirement  to  use  the  graphic  to  answer  the  questions.  In  total  nine  participants  would 
have  liked  the  opportunity  for  even  more  interaction  with  partnering  teams.  Rather  than  simply 
sharing  and  discussing  their  responses  to  problems,  they  would  have  liked  more  opportunities  to 
solve  problems  together.  It  was  generally  agreed  that  participants  should  be  provided  with  read- 
ahead  materials  that  outline  the  purpose  of  the  exercise  and  provide  background  to  the  conditions 
of  the  scenarios  and  that  participants  should  be  given  more  time  to  consider  the  problems  and 
come  up  with  solutions. 

A  total  of  seven  participants  commented  that  scenarios,  roles  and  requirements  should  be 
pitched  at  the  correct  level  for  the  participants,  and  moreover  that  the  requirements  should  be 
matched  to  the  participant's  role  within  the  scenario.  A  somewhat  related  issue  is  the  desire  for 
the  scenarios  and  requirements  to  match  the  MDMP  and  military  doctrine  more  closely. 

Finally,  participants  were  asked  to  provide  any  additional  comments  they  would  like  to 
make  about  the  training.  The  comments  were  focused  around  four  main  issues: 

•  The  need  for  greater  clarity  about  the  objectives  of  the  exercise. 

•  The  need  to  set  the  dilemmas  at  a  level  appropriate  to  the  rank  and  role  of  the  participants 

and  the  role  they  are  being  asked  to  play  in  the  scenario. 

•  The  need  for  a  greater  level  of  interactivity  with  partners  in  problem  solving. 

•  Usefulness  of  the  training. 

Discussion  Results.  Participants  provided  feedback  on  six  main  issues: 

•  The  training  tool. 

•  The  scenarios. 

•  Participant  requirements. 

•  The  facilitation  process. 
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•  The  relevance  of  the  training. 

•  Overall  reactions. 

Regarding  the  tool,  it  was  generally  felt  that  there  was  limited  interaction  with  it  and 
particularly  with  the  graphics  presented.  A  majority  of  participants  would  have  liked  more 
opportunity  to  engage  in  problem  solving  processes  that  involved  greater  interactivity  with  the 
tool  and  the  graphics.  A  majority  of  participants  felt  that  the  requirements  supported  the  learning 
process.  However  a  minority  stated  that  they  would  have  liked  more  time  to  consider  the 
problems  presented,  that  requirements  could  have  been  more  specific,  and  that  they  should  have 
had  more  interaction  with  partners  in  the  problem  solving  process. 

The  scenarios  were  generally  perceived  to  be  authentic  and  relevant;  however,  two 
participants  preferred  that  they  be  closer  to  military  doctrine.  In  total,  three  command  and 
control  (C2)  anomalies  were  highlighted  which  were  corrected  in  the  final  version  of  the 
scenarios.  Two  participants  recommended  the  use  of  actual  incidents  in  the  scenarios.  It  was 
widely  agreed  that  the  experienced  facilitator  added  value  to  the  process  by  introducing  other 
perspectives  and  broadening  those  perspectives.  There  was  general  agreement  regarding  the 
relevance  of  the  training  for  Soldiers  preparing  for  deployment  to  a  coalition  operational 
environment.  However,  there  was  a  mix  of  beliefs  about  w  hether  previous  coalition  experience 
was  necessary  or  desirable  for  participation  in  the  training.  About  half  of  the  participants  felt 
that  this  training  is  only  appropriate  for  those  with  previous  coalition  experience,  and  that  it  is 
not  suitable  as  baseline  training.  Conversely,  half  of  the  participants  felt  that  anyone  on  a  BDE 
or  BN  staff  would  benefit  from  this  training. 

Overall,  the  training  provided  a  good  opportunity  to  problem  solve  and  exchange  ideas 
with  coalition  partners.  A  majority  of  participants  stated  that  the  exercise  helped  them  to 
understand  not  only  the  partner’s  point  of  view,  but  also  how  the  partner  might  view  their 
actions. 

Level  2:  Learning:  Situation  Awareness  (S.4)  Exercise 

The  SA  Exercise  was  used  to  assess  whether  learning  was  taking  place  as  a  result  of  the 
training.  The  first  SA  Exercise  was  completed  before  any  discussion  of  collaborative  problem 
solving  took  place  between  participants.  The  second  and  third  SA  Exercises  took  place  after 
participants  engaged  in  discussions  and  collaborative  problem  solving  activities.  Below  are 
participants’  responses  from  the  SA  Exercise  from  Situation  1  (collated  across  all  three 
scenarios). 

Responses  from  SA  calibration  exercise  for  Situation  1 

1 .  Q  What  are  your  top  three  concerns  and  w  hy? 

A  Tactical,  infrastructure,  task  constraints,  resources,  personnel,  gaining  control  of 
situation,  budget,  priorities,  reconnaissance,  intelligence,  situation  report 
(SITREP),  threat,  timing,  location. 

2.  Q  What  are  your  assumptions  about  this  situation? 
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A  Capabilities,  supplies,  insurgent  actions,  security  level,  risk  level,  current  state 
command  and  control,  posture. 

3.  Q  What  are  your  requests  for  information  from  the  other  team? 

A  Capabilities,  only  logistical  support,  resources,  none,  enemy/insurgenl  activity, 
mission  and  tasks,  command,  control,  communications,  computers,  intelligence, 
surveil lance,  and  reconnaissance  (C4ISR)  compatibility,  adjacent  commands. 

Below  are  participants"  responses  from  the  SA  Exercise  from  Situation  2  and  3  (collated 
across  all  three  scenarios). 

1 .  Q  What  are  your  top  three  concerns  and  why? 

A  Threat  assessment,  engaging  leaders,  what  additional  assets  will  other  team 
require,  clarify  responsibilities  with  partnering  organization,  what  is  the  effect  of 
own  posture  on  climate,  coalition  cohesion,  who  is  the  lead  agency,  developing  a 
joint  plan,  any  competing  priorities,  w'ho  has  primacy,  consider  complicating 
political/coalition  factors. 

2.  Q  What  are  your  assumptions  about  this  situation? 

A  Climate,  likely  response  to  actions,  likely  Course  of  Action  (CoA)  of  partners 

3.  Q  What  are  your  requests  for  information  from  the  other  team? 

A  Considering  ROE.  coordinating  with  other  nations  Information  Operations  (10) 
and  public  affairs,  requests  for  information  about  CoA,  question  other  team’s 
priorities,  question  national  caveats,  requests  for  additional  information,  anticipate 
local  reactions,  anticipate  partner  reactions/implications,  other  actors/other 
involved  parties. 

The  results  of  the  first  SA  Exercise  suggests  that  respondents  are  already  taking  partners’ 
resources  and  capabilities  into  consideration  in  their  planning  and  decision  making.  However, 
examination  of  the  second  and  third  SA  Exercises  suggests  that  participants  are  considering  a 
broader  range  of  issues  and  have  greater  awareness  of  partners’  perspective  in  the  second  and 
third  vignettes  -  after  discussion  and  collaborative  problem  solving  activities  have  taken  place. 
For  example,  in  the  second  and  third  vignettes  participants  raised  the  following  issues  that  were 
not  considered  in  the  first  vignette:  estimating  what  assets  partners  may  require,  clarifying 
responsibilities  with  partnering  organizations,  coalition  cohesion,  coordinating  with  other 
nations,  and  anticipating  partner  reactions/implications.  These  findings  suggest  that  the 
discussion  sessions  and  collaborative  problem  solving  process  promoted  a  greater  awareness  of 
coalition  issues  and  how  to  respond  to  them. 

Level  3:  Behavior:  Behavioral  Markers 

Behavior,  as  defined  here,  refers  to  transfer  of  behaviors  learned  to  the  job  or  a  simulated 
environment.  While  we  were  not  assessing  team  behaviors  in  an  environment  separate  from  the 
training  evaluation,  we  did  examine  teamwork  performance  during  training  and  assessed  the 
utility  of  a  tool  for  assessing  teamwork.  Such  a  tool  could  be  used  to  assess  teamwork  prior  to 
and  after  training  or  across  training  scenarios.  Evaluators  at  each  site  during  the  evaluation  rated 
team  behaviors  using  a  behavioral  marker  system.  The  evaluators  first  indicated  whether  a 
particular  element  of  behavior  was  observed.  Then,  following  the  end  of  the  exercise,  the 
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evaluator  made  a  judgment  about  the  team’s  performance  on  each  of  the  five  dimensions.  The 
mean  ratings  given  by  six  observers  across  each  category,  on  a  scale  of  1  (very’  poor)  to  5  (very 
good),  are  shown  in  Table  3.  The  ratings  table  suggests  that  teams  performed  adequately  on  all 
dimensions  of  teamwork  assessed  in  this  exercise.  The  mean  ratings  for  team  conceptual  level 
were  given  the  highest  ratings,  cooperation  within  teams  was  rated  the  lowest,  however,  no 
statistically  significant  difference  was  found  to  exist. 

Table  3 

Mean  Observer  Ratings  for  Five  Behavioral  Categories 


Category 

Mean  ratings  across  three  scenarios 

Team  identity 

4.1 

Team  conceptual  level 

4.3 

Team  self  monitoring 

4.0 

Decision-making 

4.1 

Cooperation 

3.6 

It  should  be  noted  that  this  is  a  within  team  rating  of  performance.  The  results  do  not 
show  whether  there  was  an  improvement  in  teamwork  between  teams. 

Level  4:  Organization 

The  ultimate  aim  of  the  highest  level  of  training  evaluation  is  to  produce  a  difference  at 
the  organizational  level.  No  specific  measure  of  organizational  level  change  was  developed  for 
this  project.  The  small  numbers  of  participants  and  limited  scope  of  this  evaluation  prohibited 
any  such  endeavor,  therefore,  it  is  difficult  to  assess  whether  organization  level  change  occurred 
as  a  result  of  this  training.  However,  the  results  of  the  SA  calibration  exercise  show  that 
participants  are  considering  a  broader  range  of  issues  and  have  greater  awareness  of  partners' 
perspective  following  discussion  and  collaborative  problem  solving.  Moreover,  the  results  show 
consistent  change  in  the  right  direction  across  the  three  vignettes  for  all  three  scenarios.  These 
results  show  that  organizational  level  change  is  possible  with  wider  implementation  of  the 
training.  However,  specific  measures  to  assess  such  a  change  would  have  to  be  developed. 

Lessons  Learned 

The  training  effort  was  based  on  the  theoretical  foundation  of  previous  training 
development  in  the  area  of  advanced  cognitive  training  for  tactical  thinking.  The  Commander 
TLAC  program  addresses  tactical  thinking  skills,  while  much  of  the  Decision  Skills  Training 
development  has  focused  on  tactical  decision  making.  The  TLAC  is  built  on  the  notion  of 
theme-based  training.  This  was  translated  into  the  development  of  High  Level  Focus  Areas  in 
this  project.  Decision  Skills  Training  provided  a  foundation  of  facilitation  techniques  for 
exploring  and  improving  mental  models  of  decision-making.  This  effort  explored  the  application 
of  these  successful  learning  strategies,  using  a  low  technology,  web-based,  distributed, 
synchronous  training  method.  A  number  of  lessons  learned  from  evaluation  efforts  set  the  stage 
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for  the  use  and  refinement  of  the  Global  Teams  coalition  training  developed  in  this  project  both 
in  terms  of  the  learning  strategy  and  the  training  method. 

Technology) 

The  technology  supporting  this  training  was  purposely  designed  to  be  low  tech  so  that 
participants  with  a  low  level  of  technology  available  would  still  be  able  to  access  the  tool  and 
participate  in  the  training,  Specifically,  the  tool  was  designed  so  that  either  a  dial-up  system  or  a 
DSL  connection  could  be  used  to  gain  access.  Nonetheless,  significant  problems  were 
encountered  w  ith  the  technological  side  of  this  training.  First,  firewalls  and  blocks  on  streaming 
communications  created  considerable  access  problems  for  all  participants,  both  national  and 
international.  A  significant  time  investment  was  required  to  resolve  these  issues  and  involved 
the  coordination  of  computing  technicians  from  partnering  nations.  Even  the  telephone 
connection  created  some  problems.  Internationa!  dialing  codes  and  speaker  phone  problems 
created  delays  in  the  training,  and  there  were  significant  costs  for  international  telephone  charges 
as  well  as  potential  problems  for  users  to  obtain  permission  to  make  the  calls  to  support  training. 
A  functionality  which  has  the  ability  to  overcome  this  latter  problem  is  the  introduction  of  voice 
over  internet  protocol  (VOIP)  to  the  tool.  We  originally  did  not  add  this  function  to  conform  to 
our  design  criteria  of  “low  technology.”  This  functionality  increases  band  width  issues  with  the 
technology  and  means  that  a  DSL  connection  will  be  required  to  use  the  technology,  but  it  also 
eliminates  one  communication  channel  problem.  Moreover,  participants  appear  to  prefer  high- 
tech  functionality,  particularly  those  that  improve  the  real-time  interaction  with  partners. 

For  the  future,  the  tool  should  have  an  upload  function  which  would  allow  an  administrator  or 
facilitator  to  upload  new  or  existing  scenarios.  Our  goal  is  to  allow  researchers  to  add  scenarios 
for  research,  and  the  administrative  rights  of  allowing  various  facilitators  to  upload  scenarios  will 
be  the  purview  of  the  eventual  user  audience,  should  that  function  be  desired. 

The  Learning  Strategy’  and  the  Training  Audience 

There  is  a  need  to  clarify  the  intent  of  the  training  and  the  learning  objectives  ahead  of 
the  training  exercise.  Participants  need  to  have  a  context  for  the  training  so  that  they  can  situate 
themselves  to  that  learning  environment.  A  few  participants  in  the  evaluation  were 
uncomfortable  that  this  training  did  not  correspond  well  with  the  doctrinal  training  they  have 
received  over  many  years  and  the  fact  that  this  training  did  not  follow  the  same  format  as  the 
Military  Decision  Making  Process  (MDMP).  If  training  were  to  be  considered  on  a  continuum 
of  highly  structured  at  one  end  to  highly  unstructured  at  the  other  end,  the  MDMP  training 
process  would  sit  at  the  highly  structured  end  of  the  continuum  whereas  the  Global  Teams 
training  would  sit  at  the  highly  unstructured  end.  The  aim  of  the  Global  Teams  training  is  real 
time  discovery  and  the  development  of  understanding  between  participants.  Each  training 
session  will  uncover  different  aspects  of  the  problem  environment  due  to  the  different 
experiences  and  perspectives  that  participants  bring  to  the  training  environment.  Some  of  the 
discomfort  fell  by  participants  may  be  due  to  the  fact  that  the  training  presented  here  did  not 
conform  to  their  previous  experience  of  military  training.  This  training  forced  the  participants  to 
consider  problems  without  the  guiding  principles  of  doctrinal  training,  since  doctrine  is  different 
across  coalition  nat  ions.  A  minority  of  participants  were  uncomfortable  with  this  level  of 
flexibility  in  the  training  process.  Discomfort  was  also  caused  by  the  fact  that  there  may  have 
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been  a  mismatch  between  participants'  expectations  for  the  training  and  the  training  they  actually 
received.  To  this  end,  facilitators  should  clarify  what  the  training  is  and  what  it  is  not.  In 
particular,  it  should  be  emphasized  that  this  training  is  not  intended  to  be  an  extension  of  MDMP 
training,  and  it  is  not  an  exercise  in  detailed  planning.  Rather,  the  aim  of  the  training  is  to 
introduce  participants  to  coalition  challenges,  provide  them  with  a  safe  environment  to  explore 
differences  in  culture,  approach,  stance,  and  style,  and  help  them  to  de-center  and  take  on 
another's  point  of  view  when  engaging  in  collaborative  problem  solving. 

Emphasis  should  be  placed  on  the  problem  solving  and  coordination  aspects  of  the 
exercise  rather  than  on  the  accuracy  of  the  facts  or  participants'  experiences  of  how  things 
operate  in  theater.  Facilitation  can  focus  the  participants  away  from  disagreement  with  the 
scenario  and  onto  the  issues  of  collaboration  by  emphasizing  that  this  is  a  notional  situation 
based  in  a  realistic  part  of  the  world.  Changes  should  be  made  if  the  facts  interfere  with 
judgments  and  decisions  the  participants  are  making,  or  if  the  facts  do  not  reflect  the  nature  of 
the  actual  types  of  decisions  and  situations  confronted  by  experts  in  the  field. 

The  evaluation  process  highlighted  that  the  greatest  benefit  came  from  the  interaction  of 
partners  over  problem  solving  and  discussion  of  courses  of  action  or  plans  for  action.  It  is  clear 
that  this  learning  strategy  and  training  method  should  capitalize  on  the  opportunity  for 
interaction  as  much  as  possible  by  allowing  partners  to  interact  not  only  over  discussion  but  also 
over  problem  solving  tasks.  This  could  involve  allowing  participants  the  opportunity  to  set  their 
own  learning  agenda  by  allowing  them  to  figure  out  how  to  coordinate  most  effectively  or  to 
figure  out  what  information  or  resources  they  need  from  each  other  or  how  the  problem  should 
be  solved  or  tackled.  For  example,  one  suggestion  during  the  evaluation  was  to  give  teams  from 
each  country  different  information  to  bring  to  the  problem  (relevant  to  what  they  might  actually 
know  in  the  situation)  to  force  collaboration. 

Essentially,  the  more  collaboration  and  interaction  participants  engage  in,  the  better.  The 
problem  solving  process  would  allow  them  not  only  to  work  together  on  solving  the  problem  at 
hand,  but  also  provide  the  opportunity  for  participants  to  collaborate  over  defining  the  problem 
and  the  role  of  each  nation  in  resolving  it.  Deliberating  courses  of  action  and  assumptions 
behind  them  follows  the  facilitation  techniques  of  Decision  Skills  Training.  Honing  the 
challenges  in  the  scenarios  and  getting  the  right  mix  of  participants  working  together  at  the  right 
level  will  allow  researchers  to  better  develop  the  facilitation  techniques  to  increase  interaction. 

There  was  a  definite  reluctance  on  the  part  of  participants  to  disagree  and  contradict 
partners’  assumptions,  plans,  or  goals.  This  may  be  due  to  the  fact  that  military  culture 
transcends  national  cultural  differences.  It  may  be  due  to  the  fact  that  English-speaking  partners 
are  more  similar  than  different,  or  it  may  be  due  to  a  mutual  respect  for  partners  of  each  others’ 
viewpoint.  We  identified  a  number  of  conflicting  assumptions  in  our  data  collection,  and  we 
expect  there  is  more  difference  among  English-speaking  partners  than  was  shown  in  the 
evaluation.  It  is  also  likely  that  the  participants  in  the  evaluation  were  not  equally  matched  in 
terms  of  experience.  Some  partners  had  more  to  offer  to  the  training  experience  than  others. 
Fruitful  explorations  of  the  concepts  captured  in  the  HLFA  are  not  possible  without  some 
background  experience  in  these  areas.  Alternately,  providing  more  specific  information  in  the 
scenarios  to  highlight  areas  of  difference  and  create  more  of  a  dilemma  would  probably  benefit 
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less-experienced  participants  (such  as  those  who  soon  will  be  going  into  coalition  operations  for 
the  first  time).  Nonetheless,  it  is  important  to  get  at  the  differences  and  to  get  participants  to 
state  their  stance  clearly,  even  if  it  disagrees  with  a  partner’s  stance. 

An  important  lesson  for  the  evaluation  process  is  the  need  for  the  training  to  be  set  at  the 
appropriate  level.  In  other  words,  the  scenario  itself,  the  requirements,  participants'  actual  rank, 
and  their  assumed  role  in  the  scenario  all  should  be  in  synch.  If  any  of  these  elements  are  out  of 
place,  participants  may  be  reluctant  to  make  decisions  at  a  level  that  is  incongruous  with  their 
role  or  rank.  Specifically,  BN  and  BDE  level  staff  operating  at  the  Multi-National  Division  level 
should  deal  with  tactical  issues.  Combined  Joint  Task  Force  staff  should  be  tasked  to  deal  with 
operational  levels  and  strategic  issues.  While  the  tactical  and  operational  levels  of  assessment 
and  action  are  beginning  to  blend,  and  more  complex  decisions  are  being  made  at  lower  echelons 
than  before,  training  participants  still  need  to  feel  situated  in  their  role  and  level  of  responsibility 
to  support  discussion  and  collaboration. 

An  experienced  military  or  former  military  facilitator  is  a  crucial  component  of  this 
training  methodology.  Facilitators  with  a  military'  background  provide  an  element  of  authenticity 
to  the  training  experience.  They  can  answer  questions  and  field  participant  questions  better  than 
non-military  facilitators;  they  can  add  to  the  discussion  and  suggest  alternative  courses  of  action; 
and  they  can  feed  off  participant  questions  to  suggest  alternative  interpretations  of  events.  For 
greater  success,  these  facilitators  need  to  work  with  a  researcher  or  trainer  to  understand  the 
intent  of  the  facilitating  for  advanced  cognitive  training,  or  possess  some  background  in  this 
level  of  training  already. 


Online  Facilitation  of  Collaborative  Learning 

In  facilitating  the  problem  solving  process,  it  is  important  for  the  facilitator  to  encourage 
participants  to  think  (out  loud)  about  how  they  are  thinking  about  the  problem,  as  opposed  to 
simply  thinking  about  solutions.  In  particular,  facilitators  should  expose  participants’ 
assumptions  about  the  problem  situation  and  how  these  may  differ  across  nations.  Bringing 
assumptions  to  light  during  the  evaluation  training  sessions  was  perhaps  the  most  effective  part 
of  facilitation  in  this  project.  The  focus  for  continued  development  of  facilitation  is  to  get  the 
participants  into  actual  collaboration  over  problems  while  not  letting  the  technical  problems 
become  the  focus.  The  participants  must  continue  to  examine  their  assumptions  as  the 
collaboration  is  ongoing. 

Tool  functionality  must  be  tied  into  the  learning  strategy.  In  other  words,  the  scenarios, 
the  graphics  that  support  them,  the  participant  requirements  and  how  they  are  answered,  and  the 
facilitation  process  must  all  be  synchronized  with  the  functionality  of  the  tool.  In  this  way  the 
functionality  actually  supports  and  enhances  the  learning.  The  training  method  and  learning 
strategy,  as  we  are  using  the  terms,  are  linked  in  online  collaboration.  Tools  should  support 
facilitators  at  different  levels  of  expertise,  guiding  the  processes  as  needed.  While  such  tools  are 
available  more  readily  for  operational  collaboration  (such  as  meeting  support),  a  similar 
technology  development  is  not  seen  in  education.  However,  attention  to  the  role  and  processes 
of  online  facilitation  is  found  in  the  educational  research  community.  Technology  for  online 
collaborative  learners  should  support  the  roles  and  processes  being  developed  in  the  educational 
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literature.  Current  research  into  the  roles  and  responsibilities  of  the  online  facilitator  can  serve  as 
a  basis  for  describing  the  roles  and  responsibilities  of  the  facilitator  for  the  Global  Teams  tool 
(Berge,  1995;  Hootstein,  2002).  This  literature  serves  as  a  basis  to  clarify  the  facilitator’s  role 
and  need  for  technology  support,  but  refinement  for  this  particular  level  of  cognitive  learning  and 
training  audience  will  still  be  needed. 


Conclusions 

With  the  ever-increasing  involvement  in  coalition  operations,  it  is  essential  to  begin 
looking  at  training  and  readiness.  The  training  that  currently  exists  is  limited  in  scope,  depth, 
and  breadth,  and  tends  to  be  mission-specific  and  region-specific.  This  report  describes  the 
research  and  development  of  a  web-hosted,  scenario-based  training  tool  for  coalition  forces.  The 
analysis  of  interviews  with  experienced  coalition  forces  revealed  a  number  of  key  focus  areas 
and  awareness  points  which  were  central  to  the  training  tool.  Scenarios  were  developed  to  force 
participants  to  engage  in  problem  solving  in  these  areas,  to  discuss  and  grapple  with  them.  The 
evaluations  highlighted  two  key  findings:  first,  that  the  scenario  challenges  need  to  support 
collaborative  problem  solving  and  second,  that  the  training  tool  should  support  the  collaborative 
process.  One  way  to  achieve  the  former  is  to  provide  participants  with  challenges  that  are  as 
specific  as  possible  so  that  participants  are  forced  to  coordinate  and  collaborate  in  solving  them. 

A  way  to  modify  or  enhance  the  existing  process  would  be  to  give  teams  differing  information 
and  tasks  so  that  they  have  to  come  together  to  achieve  one  larger  task  or  challenge.  For 
example,  each  team  could  be  tasked  with  different  information  and  area  of  responsibility  towards 
a  mission.  The  challenge  should  require  them  to  come  together  to  discuss  and  exchange 
recommendations  based  on  prior  knowledge  and  tasking. 

The  evaluations  highlighted  a  number  of  factors  which  can  impact  the  effectiveness  of 
the  training  including:  the  importance  of  clarifying  the  intent  of  the  training  and  the  learning 
objectives  in  advance,  the  need  to  emphasize  the  problem  solving  and  coordination  aspects  of  the 
exercise  rather  than  the  accuracy  of  the  facts,  the  merit  in  capitalizing  on  the  opportunity  for 
interaction  as  much  as  possible  by  allowing  partners  to  interact  not  only  over  discussion  but  also 
over  problem  solving,  the  need  for  the  training  to  be  set  at  the  appropriate  level,  the  value  of  an 
experienced  facilitator,  and  the  significance  of  tying  tool  functionality  to  the  learning  strategy. 
Overall,  the  training  was  shown  to  improve  participants’  awareness  of  and  ability  to  respond  to 
the  key  themes  of  coalition  coordination.  The  tool  may  provide  an  easy-to-implement  and  cost- 
efficient  means  for  coalition  partners  to  train  in  a  distributed  environment,  on  short  notice,  prior 
to  deploying  to  the  operating  environment. 

This  research  effort  was  designed  to  test  the  viability  of  the  collaborative  distributed 
training  method  as  a  means  to  train  distributed  teams  in  coalition  problem  solving  and 
coordination.  The  products  from  this  effort  include  a  web-based  training  tool,  three  pre¬ 
developed  scenarios  and  three  accompanying  facilitation  guides.  The  tool  incorporates  both  a 
database  and  server-based  filing  structure  that  contains  the  pre-developed  scenarios.  Library 
scenarios  can  consist  of  up  to  four  total  graphics,  four  situations  (including  a  background),  rules 
of  engagement,  and  a  facilitator's  guide.  This  tool  is  available  from  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  to  parties  interested  in  hosting  such 
training. 
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Appendix  A 

Global  Teams  Training  Tool:  Primary  Supported  Functionalities 


Global  Teams  Training  Tool;  Primary  Supported  Functionalities 

•  Login  Restriction 

At  the  introduction  screen,  access  is  restricted  to  only  those  users  who  have  recognized 
Player  IDs. 

•  Dynamic  Player  Notes  Area 

At  the  introduction  screen,  a  dynamic  note  area  allows  facilitators  to  leave  initial 
guidance  to  players,  such  as  timing  of  the  next  session  and  specifics  on  which  vignette  is 
to  be  conducted. 

•  Dynamic  Library  of  Vignettes 

The  tool  incorporates  both  a  database  and  server-based  filing  structure  that  contains  three 
pre-developed  vignettes.  Library  vignettes  can  consist  of  up  to  four  total  graphics,  four 
situations  (including  a  background),  rules  of  engagement,  and  a  facilitator's  guide.  The 
tool  also  stores  vignette  identifiers  to  support  the  recognition/selection  of  desired 
vignettes.  After  initially  logging  in,  the  tool  presents  summary  information  (title, 
descriptive  summary,  creation  date,  and  author)  for  each  vignette  in  the  database,  and 
allows  the  user  to  scroll  through  the  summaries  and  select  the  vignette  of  interest. 

•  Customized  Imerface/Functionality  Based  on  User  Login 

Users  who  log  in  as  facilitators  have  access  to  all  player  responses  as  they  are  entered, 
whereas  players  representing  the  countries  and  organizations  can  only  view'  their  own 
responses,  unless  the  facilitator  shares  information.  Further,  the  facilitators/observers 
have  access  to  an  onscreen  chat  area  only  visible  to  them.  This  feature  supports 
administrative  chats  (e.g.,  how  to  proceed,  observations  of  a  session,  when  to  take  breaks, 
etc.)  that  would  otherwise  distract  the  players, 

•  Ability  to  Upload  New  Vignettes 

An  innovative  and  essential  feature  of  the  tool  is  its  ability  to  support  the  uploading  of 
new  vignettes.  Users  wrho  are  logged  in  as  administrators  have  the  option  of  uploading 
their  own  vignettes,  including  text,  graphics,  supporting  summary,  and  facilitation 
information.  This  allows  for  an  ever-growing  body  of  vignettes  that,  once  uploaded,  can 
be  conducted  by  any  experienced  facilitator. 

•  Collaborative,  Synchronous  Graphics  Manipulation 

As  the  tool  presents  graphics  associated  with  vignettes  during  a  session,  all  users  have  the 
ability  to  superimpose  images  and  text  over  the  graphics.  Users  can  choose  to  draw  line 
segments,  directional  arrows,  boxes,  circles,  and  even  freeform  sketches  for  all  other 
users  see.  Users  can  also  enter  freeform  text  and/or  textboxes  and  customize  the  colors  of 
any  of  their  selections.  The  tool  allows  users  to  individually  select,  move,  or  delete 
individual  drawing  items,  or  clear  the  entire  screen  with  the  push  of  a  button. 

•  Collaborative,  Synchronous  Whitespace 

In  addition  to  the  graphics  drawing,  the  Global  Teams  tool  provides  a  shared  white  space 
where  players  and  facilitators  can  sketch  their  own  maps.  This  supports  more  freeform 
play  of  a  vignette,  where  a  facilitator  can  create  a  new  situation  and  have  players  respond. 

•  Private  Response  Forms 

Players  have  access,  depending  on  their  login,  to  specific  response  forms.  These  forms 
are  not  visible  to  other  players  and  provide  a  semi -private  area  for  a  player/team  to  work 
out  and  record  answers  to  vignette  questions  or  requirements.  Facilitators  have  access  to 
all  the  response  forms  so  they  can:  1 )  assess  progress  online,  2)  determine  whether 
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individual  players/teams  require  additional  assistance  or  need  to  be  redirected.  3)  observe 
the  response  development  process,  and  4)  evaluate  relative  responses  as  they  are  being 
formed , 

•  Split-Screen  Shared  Space 

The  Shared  Space  area  of  the  tool  consists  of  two  vertically  aligned  shared  text  spaces 
where  the  facilitator  can  present  information  that  all  players  can  simultaneously  view. 

The  split-screen  feature  allows  facilitators  to  present  responses  from  different 
players/teams  to  the  same  questions  side-by-side  (if  desired),  to  support  the  comparing 
and  contrasting  of  responses. 

•  Mouse/Chat  Color  and  Name  Identification 

Often  in  collaborative,  online  environments  it  is  difficult  to  identify  who  is  doing  what 
onscreen.  For  this  reason,  the  tool  provides  name  identifiers  and  attaches  them  to  each 
user's  mouse  icon.  To  do  this,  the  player  enters  a  screen  name  that  he/she  wishes  to  use, 
and  then  his/her  pointer  icon  displays  that  name  for  other  users  to  see.  Further,  individual 
players/facilitators  can  also  customize  their  pointers  by  selecting  their  own  color.  These 
identifiers  (color  and  name)  distinguish  individual  users  in  the  main  chat  area. 

•  Chat/Facilitator  Chats 

The  tool  provides  an  area  where  any/all  users  can  have  group  chats.  The  tool  also 
provides  an  additional  area,  only  accessible  to  facilitators,  where  this  subgroup  can 
conduct  private  chats.  Again,  this  feature  supports  administrative  chats  (e.g.,  how  to 
proceed,  observations  of  a  session,  when  to  take  breaks,  etc.)  that  would  otherwise 
distract  the  players. 

•  Optional  Audio  Conferencing 

Users  have  the  option  of  utilizing  the  tool’s  streaming  audio  functionality.  If  desired, 
players  and  facilitators  can  have  their  conversations  streamed  over  the  internet, 
eliminating  the  need  for  a  phone  connection.  Players  using  this  feature  can  either  have 
the  tool  automatically  detect  when  they  are  talking  or  manually  trigger  the  sending  of 
audio  (to  reduce  extraneous  noise  or  to  ensure  privacy  in  offline  discussions).  Users  can 
transfer  between  modes  with  ease.  Onscreen  display  features  are  also  presented  to  show 
which  user  is  currently  the  active  speaker. 

•  Player  List 

The  screen  names  of  all  users  who  are  logged  onto  the  tool  are  displayed  in  a  player  list 
box.  This  feature  prevents  anonymous  observations  and  also  provides  verification  that  all 
necessary  players  are  present  for  the  vignette  session. 

•  Customized  Pop-up  Help  Screens 

To  support  novice  users,  the  tool  provides  help  files,  customized  to  different  functional 
areas  within  the  tool.  A  help  file  is  available  to  assist  facilitators  in  uploading  new' 
vignettes,  providing  specific  guidance  in  how  to  prepare  the  required  graphic  and 
MSWord  files,  and  how  to  identify  them  through  the  tool’s  upload  interface.  Additional 
help  files  provide  guidance  in  implementing  the  main  interface  functionalities. 
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Appendix  B 
Reactions  Survey 


Reactions  Survey 


1 .  Your  current  duty  position: 


2.  Rank: 


3.  Briefly  list  any  past  experience  working  with  coalitions  and  your  job  in  those  coalitions: 


4.  How  much  value  is  there  to  training  with  coalition  partners  in  general? 

None  at  all  Very  little  Uncertain  •  Some  A  great  deal 

1  2  3  4  5 

5.  How  much  value  was  there  in  training  w'ith  coalition  partners  in  this  exercise? 

None  at  all  Very  little  Uncertain  Some  A  great  deal 

1  2  3  4  5 


6.  How  would  you  rate  the  quality  of  the  content  presented  in  this  exercise? 


Very  poor 
1 


Poor 

2 


Satisfactory 
3  ’ 


Very  good 
4 


7.  How  frequently  should  a  staff  member  participate  in  similar  training? 


Never 

1 


Once 

2 


Once  a  year 
3 


2-3  times 
a  year 

4 


Excellent 

5 


Once  a  month 
5 


8.  List  three  things  that  made  the  training  more  effective  and  explain  why. 

a. 


b. 


c. 


B-2 


9.  List  three  things  that  would  make  the  training  more  effective  and  explain  how. 

a. 


b. 


c. 


1 0.  Any  additional  comments: 


B-3 


i 


B-4 


Appendix  C 
Discussion  Questions 
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Discussion  Questions 


Reactions 

•  How  useful  did  you  find  this  training? 

•  Would  you  be  willing  to  participate  in  a  similar  exercise  six  months  from  now? 
Application 

•  How  valuable  was  it  to  have  a  group  from  another  country  participating? 

•  What  military  positions  would  benefit  most  from  this  training? 

Further  development 

•  In  your  opinion,  are  the  military  details  presented  in  the  scenario  and  the  related 
questions  authentic?  How  could  they  be  improved? 

What  did  the  facilitator  do  to  support  and  reinforce  learning  during  the  exercise?  How  could 
facilitation  be  improved? 
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Appendix  D 

Behavioral  Marker  System  -  Score  Forms 


Behavioral  Marker  System  (Elements  Score  Form) 


Team  identity 

Y/N 

Team  conceptual  level 

Y/N 

•  Defining  roles  and  functions 

•  Engaging 

•  Compensating 

•  Avoiding  micromanagement 

•  Envisioning  goals  and  processes 

•  Forecasting:  time  horizons  and  range  of 
factors 

•  Detecting  gaps  and  ambiguities 

•  Achieving  situation  assessment: 
diverging  and  converging 

Team  self  monitoring 

Team  decision  making 

•  Time  management/ workload 
management 

•  Adjusting 

•  Problem  definition  and  diagnosis 

•  Option  generation 

•  Risk  assessment  and  option  selection 

•  Outcome  review 

Cooperation 

•  Teambuilding  and  maintaining 

•  Consideration  of  others 

•  Support  of  others 

•  Conflict  solving 

Behavioral  Marker  System  (Categories  Score  Form) 


Categories 

Very  poor 

f 

> 

i 

i 

o 

O 

I 

Very  good 

Noi  Observed 

Comments 

Team  identity 

Team  self  monitoring 

Cooperation 

Team  conceptual  level 

Decision  making 
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